ASTM E2991/E2991M-17

This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the
Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
Designation: E2991/E2991M − 17
Standard Test Method for
Evaluating Response Robot Mobility: Traverse Gravel
Terrain
ThisstandardisissuedunderthefixeddesignationE2991/E2991M;thenumberimmediatelyfollowingthedesignationindicatestheyear
of original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval.
A superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
INTRODUCTION
The robotics community needs ways to measure whether a particular robot is capable of performing
specific missions in unstructured and often hazardous environments. These missions decompose into
elemental robot tasks represented individually as standard test methods and practices. The associated
testapparatusesandperformancemetricsprovideatangiblelanguagetocommunicatevaryingmission
requirements. They also enable repeatable testing to establish the reliability of essential robot
capabilities.
TheASTM International Standards Committee on Homeland SecurityApplications (E54) specifies
standard test methods and practices for evaluating such robot capabilities. These standards facilitate
comparisons across diverse models or multiple configurations of a single model. They support robot
researchers, manufacturers, and user organizations in different ways. Researchers use the standards to
understand mission requirements, encourage innovation, and demonstrate break-through capabilities.
Manufacturers use the standards to evaluate design decisions, integrate emerging technologies, and
harden developed systems. User organizations leverage the resulting robot capabilities data to guide
purchasing decisions, align deployment objectives, and focus training with standard measures of
operator proficiency. Associated usage guides describe how such standards can be applied to support
these various objectives.
The overall suite of standards addresses critical subsystems of remotely operated response robots,
including maneuvering, mobility, dexterity, sensing, energy, communications, durability, proficiency,
autonomy,logistics,safety,andterminology.Thistestmethodaddressestheroboticmobilityongravel
terrain.
1. Scope corresponding performance metric may be the number of
completed task repetitions per minute over an assigned time
1.1 The purpose of this test method is to specify the
period ranging from 10 to 30 minutes.
apparatuses, procedures, and performance metrics necessary to
quantitatively measure a teleoperated ground robot’s capability
1.4 This test method is a part of the mobility suite of ground
of traversing gravel terrain. The primary performance metric
response robot test methods, but this test method is stand-alone
for this test method shall be a robot’s possession of such a
and complete. This test method applies to ground systems
capability with a specified statistical significance level.
operated remotely from a standoff distance appropriate for the
intended mission. The system includes a remote operator in
1.2 Average rate of advance over the specified terrain shall
control of all functionality and any assistive features or
be the secondary performance metric for this test method. The
autonomous behaviors that improve the effectiveness or effi-
measure shall be calculated only when a robot under test has
ciency of the overall system.
completed a statistically-significant number of repetitions.
1.3 This test method can also be used to measure the 1.5 The apparatus, specified in Section 6, can only test a
limitedrangeofarobot’scapabilities.Whentherobothasbeen
operator proficiency in performing the specified task. The
tested through the limit or limits of the apparatus, a note shall
be associated with the results indicating that the robot’s actual
This test method is under the jurisdiction of ASTM Committee E54 on
capability may be outside of the limit or limits imposed by the
Homeland Security Applications and is the direct responsibility of Subcommittee
test apparatus. For example, the size of the gravel terrain test
E54.09 on Response Robots.
apparatus could possibly affect the acceleration of the robot
Current edition approved Sept. 1, 2017. Published October 2017. DOI: 10.1520/
E2991_E2991M-17. under test and, in turn, the resulting average rate of advance.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
E2991/E2991M − 17
1.6 Performing Location—This test method may be per- 4. Summary of Test Method
formed anywhere the specified apparatuses and environmental
4.1 A robot under test traverses from one end of the gravel
conditions can be implemented.
terrain to the other and back while alternating left and right
1.7 Units—The values stated in either SI units or inch-
turns around the pylons to complete a figure-8 path. The robot
pound units are to be regarded separately as standard. The
is credited with a fixed forward distance for each completed
values stated in each system may not be exact equivalents;
figure-8 path repetition.
therefore,eachsystemshallbeusedindependentlyoftheother.
Combining values from the two systems may result in noncon-
5. Significance and Use
formance with the standard. Both units are referenced to
5.1 Traversing on terrains with small aggregate (such as
facilitate acquisition of materials internationally and minimize
Number 8 or smaller gravel, per Specification C33/C33M)
fabrication costs.
could pose problems for ground robots because the aggregate
1.8 This standard does not purport to address all of the
may become incrementally packed into the locomotion sub-
safety concerns, if any, associated with its use. It is the
systems (such as driving sprockets, belts, chains, tire treads, or
responsibility of the user of this standard to establish appro-
track pads) leading to jamming, slippage, or other failures, and
priate safety, health, and environmental practices and deter-
thus adversely affecting a robot’s mobility. This test method
mine the applicability of regulatory limitations prior to use.
addresses aforementioned issues of mobility.
1.9 This international standard was developed in accor-
dance with internationally recognized principles on standard- NOTE 1—Larger-sized gravel might not be as easily packed into robotic
locomotion subsystems but might present different types of mobility
ization established in the Decision on Principles for the
challenges such as angular, rough, sharp, or broken aggregate pieces
Development of International Standards, Guides and Recom-
interfering with wheels, tracks, or other types of locomotion mechanisms.
mendations issued by the World Trade Organization Technical
These issues are out of the scope of this test method.
Barriers to Trade (TBT) Committee.
5.1.1 Small gravel based terrains are non-rigid in nature and
could cause a robot to turn-in-place or dig-in when the robot is
2. Referenced Documents
negotiating a tight turn. Certain robotic locomotion mecha-
2.1 ASTM Standards:
nismsmightbedesignedforothermobilitypurposesandmight
C33/C33M Specification for Concrete Aggregates
notcreatesufficienttractionagainstthespecifiedgravelterrain.
D5821 Test Method for Determining the Percentage of
As such, extensive testing within this type of terrain may
Fractured Particles in Coarse Aggregate
expose robot design or reliability issues and lead to field
E2521 Terminology for Evaluating Response Robot Capa-
maintenance or repair.
bilities
5.1.2 The gravel traverse capabilities could be affected by
E2592 Practice for Evaluating Response Robot Capabilities:
additional factors such as the weight and its distribution,
Logistics: Packaging for Urban Search and Rescue Task
ground contact areas, and control schemes for the robot. As
Force Equipment Caches
such, extensive testing within this type of terrain may also lead
E2803 Test Method for Evaluating Emergency Response
to innovations in robot design.
Robot Capabilities: Mobility: Confined Area Obstacles:
Inclined Planes
5.2 Key features of response robots are that they are
remotely operated from safe standoff distances, deployable at
3. Terminology
operational tempos, capable of operating in complex
3.1 Definitions: environments, sufficiently hardened against harsh
3.1.1 The following terms are used in this test method and
environments, reliable and field serviceable, durable or cost-
are defined in Terminology E2521: abstain, administrator or effectively disposable, and equipped with operational safe-
testadministrator,emergencyresponserobotorresponserobot,
guards.As such, a major advantage of using robots in response
fault condition, operator, operator station, remote control, operations is to enhance the safety and effectiveness of
repetition, robot, teleoperation, test event or event, test form,
responders or soldiers.
test sponsor, test suite, testing target or target, testing task or
5.3 This test method aligns user expectations with actual
task, and trial or test trial.
capabilities to understand the inherent capability trade-offs in
3.1.2 The following terms are used in this test method and
deployable systems at any given cost. For example, a design
are defined in ALFUS Framework Volume I: autonomous,
issue of the number of batteries to be packed on a robot could
autonomy,levelofautonomy,operatorcontrolunit(OCU),and
affect desired weight, endurance, or cost.Appropriate levels of
semi-autonomous.
understanding can help ensure that requirement specifications
are articulated within the limit of current capabilities.
For referenced ASTM standards, visit the ASTM website, www.astm.org, or
5.4 This test method provides a tangible representation of
contact ASTM Customer Service at service@astm.org. For Annual Book of ASTM
essential robot capabilities with quantifiable measures of per-
Standards volume information, refer to the standard’s Document Summary page on
the ASTM website.
formance. When considered with other related test methods in
NIST Special Publication 1011-I-2.0 Autonomy Levels for Unmanned Systems
the suite, it facilitates communication among communities of
(ALFUS) Framework Volume I: Terminology, Version 2.0. Available from National
robotusersandmanufacturers.Assuch,thistestmethodcanbe
Institute of Standards and Technology (NIST), 100 Bureau Dr., Stop 1070,
Gaithersburg, MD 20899-1070, http://www.nist.gov. used to:
E2991/E2991M − 17
5.4.1 Inspire technical innovation and guide manufacturers 6. Apparatus
towardimplementingcombinationsofcapabilitiesnecessaryto
6.1 The test apparatus is a fixed-size gravel terrain through
perform essential mission tasks.
which the robot has to traverse (see Figs. 1 and 2 for
5.4.2 Measure and compare essential robot capabilities.
illustrations).
This test method can establish the reliability of the system to
6.1.1 The terrain size may be scaled to provide for various
perform specified tasks, highlight break-through capabilities,
levels of mobility constraints for robots under test depending
and encourage hardening of developmental systems.
on testing requirements. Three typical terrain sizes are nomi-
5.4.3 Inform purchasing decisions, conduct acceptance
nally 3.6 m [12 ft] long by 1.2 m [4 ft] wide, 7.2 m [24 ft] long
testing, and align deployment objectives with statistically
by 2.4 m [8 ft] wide, and 14.4 m [48 ft] long by 4.8 m [16 ft]
significant robot capabilities data captured through repeated
wide, respectively.Atest sponsor is authorized to specify other
testing and comparison of quantitative results.
test apparatus sizes to suit particular deployment requirements.
5.4.4 Focus operator training and measure proficiency as a
repeatable practice task that exercises actuators, sensors, and
NOTE 2—Test Method E2803 and this test apparatus can be combined
to form a gravel test terrain on an incline.
operator interfaces. The test method can be embedded into
training scenarios to capture and compare quantitative scores
6.1.2 Each of the terrains is fully covered with packed
even within uncontrolled environmental variables. This can
Number 8 aggregate (with nominal size of 2.36 to 9.5 mm, or
help develop, maintain, measure, and track very perishable
0.09 to 0.37 in.), as specified in Specification C33/C33M, for a
skillsovertimeandenablecomparisonsacrosssquads,regions,
minimal depth of 15 cm [6 in.], and evenly spread throughout
or national averages.
the apparatus floor. The gravel shall be dry, both visibly and to
5.5 Although this test method was developed for response the touch within the entire apparatus, including from the
robots, it may be applicable to other domains. Different user surface through the bottom.
communities can set their own thresholds of acceptable per-
6.1.3 Each terrain is recommended to be fully surrounded
formance within the test method for various mission require-
by walls that are nominally 1.2 m [4 ft] tall and are typically
ments.
built out of plywood or oriented strand boards (OSB). Lumber
with a nominal cross-section of 5 by 25 cm [2 by 10 in.] could
5.6 It is recommended that users of this test method con-
alsobeusedtosurroundtheterrainandhelpfurthercontainthe
sider their particular robot requirements when interpreting the
gravel.
test results. The capability evaluated in this test method alone
6.1.4 The ends of each of the terrains have end zones that
shall be interpreted according to the scope of this test method
are half as long as the width of the terrain and feature walls
and shall not be considered as an overall indication of the
completely painted with alternating black-and-white, vertical,
capability of the robot’s mobility system nor of the entire
and nominally 30 cm [12 in.] wide stripes.
roboticsystem.Asingletestmethodonlycapturesthespecified
single aspect of a robot’s capabilities. A more complete 6.1.5 Fourpylonsdefineafigure-8traversepath(seeFig.2).
The two timer pylons are placed at a distance from each other
characterization of a robot’s capabilities requires test results
from a wider set of test methods. equal to the width of the terrain and centered between the end
FIG. 1 Confined Area Terrains: Gravel; Three Sizes
E2991/E2991M − 17
FIG. 2 Confined Area Terrains: Gravel; Figure-8 Path
zones and between the sidewalls. The two outer pylons are is specified as nominally 0.1 lux. The dark test condition is not
placed at the edge of each end zone and centered between the specified to be lower than 0.1 lux because of the implementa-
sidewalls. Pylons that are typically used for general traffic
tion cost concerns. 0.1 lux is dark enough to require the robot
guidance and have nominal heights of at least 30 cm [1 ft] and
tohaveonboardillumination—atestrobotcapability—inorder
nominalcross-sectiondiametersofatmost30cm[1ft]shallbe
to perform the task. However, it is recognized that actual
acceptable for use in this test method.
operational environments may be darker.
6.2 In terms of lighting, the lit test condition is specified as
NOTE 3—When this test apparatus is implemented in an International
indoors (with furnished lighting), typically measured at nomi-
Organization for Standardization (ISO) freight container, the 0.1 lux
nally 150 to 300 lux or outdoors (in the daylight), typically
darkness could be achieved by turning off all the inside lighting sources
measured at up to 1000 lux, nominally. The dark test condition
E2991/E2991M − 17
and covering the entrance of the container entirely with light-blocking
maintenance tasks (for example, battery change). The system
drapes.
configuration shall remain the same for all relevant tests to
6.3 Appropriate measuring devices shall be used to measure enable direct comparison of performance and to identify
the robot’s performance within an apparatus. capability trade-offs between different configurations. Any
number of identified system configurations can be subjected to
NOTE 4—A stopwatch may be used to measure the time and an
testing.
electronic beam-breaking sensor could be used to count the number of
repetitions. Video captured with four simultaneous feeds is highly
8.3 Determine the number of required repetitions if measur-
recommended for recording the tests. The four simultaneous video feeds
ing system capabilities.
are: an overall view of the apparatus, a detailed view of the robot
8.3.1 Test trials shall produce enough successful repetitions
performing the task, a view of the OCU’s display showing the view from
therobot’sonboardcamera,andaviewoftheoperator’shandscontrolling to demonstrate the reliability of the system or operator neces-
therobot.Thistypeofrecordhelpsverifytestresultsandenhanceoperator
sary for the envisioned missions. The higher the ratio of
training.
successful repetitions to faults, the more reliable the system or
operator. The more repetitions completed with that ratio, the
7. Hazards
more confidence may be placed in that reliability. The calcu-
7.1 In addition to 1.8, users of this test method shall also
lated reliability and confidence levels can be determined from
address equipment preservation and human-robot coexistence
statistical tables. Some missions may require higher reliability.
concerns. Safety setups such as belays and containment walls
Others may be more resilient to failure and can accommodate
shall be used when there are such concerns. Environmental
lower reliability. A test trial of 30 repetitions or more is
conditions, such as high or low temperatures, excessive
recommended to establish a system’s capability. Operator
moisture, and rough terrains can be stressful, exceed the
proficiency trials are typically time limited as specified in 8.4.
respective ranges within which the robot is built to properly
8.3.2 A reasonable starting threshold may be at least 80 %
operate, or damage robotic components. These conditions can
reliability with 80 % confidence. This can be achieved by
also cause unexpected robot motions that, in turn, can have
performing 30 repetitions with 27 or more successes. When 30
negative effects on the humans that are nearby or on the robot
repetitions is not feasible, this reliability and confidence may
itself.
stillbeachievedifthefirsttesttrialincludes20repetitionswith
7.2 Identify all the emergency stop (E-stop) button(s) on the
19 or more successes or 10 successful repetitions.
robot chassis and the OCU before operating or interacting with
8.3.3 Multiple trials are allowed to improve the perfor-
the robot.
manceofthesametestedrobotconfiguration.Ifthisisthecase,
the latest 30 consecutive repetitions from across the multiple
7.3 While the robot is active and the E-stop button is
trials shall be considered together when determining the ratio
disengaged, avoid:
of suc
...