ASTM E2992/E2992M-17

This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the
Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
Designation: E2992/E2992M − 17
Standard Test Method for
Evaluating Response Robot Mobility: Traverse Sand Terrain
ThisstandardisissuedunderthefixeddesignationE2992/E2992M;thenumberimmediatelyfollowingthedesignationindicatestheyear
of original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval.
A superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
INTRODUCTION
The robotics community needs ways to measure whether a particular robot is capable of performing
specific missions in unstructured and often hazardous environments. These missions decompose into
elemental robot tasks represented individually as standard test methods and practices. The associated
testapparatusesandperformancemetricsprovideatangiblelanguagetocommunicatevaryingmission
requirements. They also enable repeatable testing to establish the reliability of essential robot
capabilities.
TheASTM International Standards Committee on Homeland SecurityApplications (E54) specifies
standard test methods and practices for evaluating such robot capabilities. These standards facilitate
comparisons across diverse models or multiple configurations of a single model. They support robot
researchers, manufacturers, and user organizations in different ways. Researchers use the standards to
understand mission requirements, encourage innovation, and demonstrate break-through capabilities.
Manufacturers use the standards to evaluate design decisions, integrate emerging technologies, and
harden developed systems. User organizations leverage the resulting robot capabilities data to guide
purchasing decisions, align deployment objectives, and focus training with standard measures of
operator proficiency. Associated usage guides describe how such standards can be applied to support
these various objectives.
The overall suite of standards addresses critical subsystems of remotely operated response robots,
including maneuvering, mobility, dexterity, sensing, energy, communications, durability, proficiency,
autonomy, logistics, safety, and terminology. This test method addresses the robotic mobility on sand
terrain.
1. Scope corresponding performance metric may be the number of
completed task repetitions per minute over an assigned time
1.1 The purpose of this test method is to specify the
period ranging from 10 to 30 minutes.
apparatuses, procedures, and performance metrics necessary to
1.4 This test method is a part of the mobility suite of ground
quantitatively measure a teleoperated ground robot’s capability
of traversing sand terrain. The primary performance metric for response robot test methods, but this test method is stand-alone
and complete. This test method applies to ground systems
this test method shall be a robot’s possession of such a
capability with a specified statistical significance level. operated remotely from a standoff distance appropriate for the
intended mission. The system includes a remote operator in
1.2 Average rate of advance over the specified terrain shall
control of all functionality and any assistive features or
be the secondary performance metric for this test method. The
autonomous behaviors that improve the effectiveness or effi-
measure shall be calculated only when a robot under test has
ciency of the overall system.
completed a statistically-significant number of repetitions.
1.5 The apparatus, specified in Section 6, can only test a
1.3 This test method can also be used to measure the
limitedrangeofarobot’scapabilities.Whentherobothasbeen
operator proficiency in performing the specified task. The
tested through the limit or limits of the apparatus, a note shall
be associated with the results indicating that the robot’s actual
This test method is under the jurisdiction of ASTM Committee E54 on
capability may be outside of the limit or limits imposed by the
Homeland Security Applications and is the direct responsibility of Subcommittee
test apparatus. For example, the size of the sand terrain test
E54.09 on Response Robots.
apparatus could possibly affect the acceleration of the robot
Current edition approved Sept. 1, 2017. Published October 2017. DOI: 10.1520/
E2992_E2992M-17. under test and, in turn, the resulting average rate of advance.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
E2992/E2992M − 17
1.6 Performing Location—This test method may be per- 4. Summary of Test Method
formed anywhere the specified apparatuses and environmental
4.1 A robot under test traverses from one end of the sand
conditions can be implemented.
terrain to the other and back while alternating left and right
1.7 Units—The values stated in either SI units or inch-
turns around the pylons to complete a figure-8 path. The robot
pound units are to be regarded separately as standard. The
is credited with a fixed forward distance for each completed
values stated in each system may not be exact equivalents;
figure-8 path repetition.
therefore,eachsystemshallbeusedindependentlyoftheother.
Combining values from the two systems may result in noncon-
5. Significance and Use
formance with the standard. Both units are referenced to
5.1 Traversing on terrains with fine aggregate, such as
facilitate acquisition of materials internationally and minimize
specified in Specification C144, could pose problems for
fabrication costs.
ground robots because the aggregate may become incremen-
1.8 This standard does not purport to address all of the
tally packed into the locomotion subsystems (such as driving
safety concerns, if any, associated with its use. It is the
sprockets, belts, chains, tire treads, or track pads) leading to
responsibility of the user of this standard to establish appro-
jamming, slippage, or other failures, and thus adversely affect-
priate safety, health, and environmental practices and deter-
ing a robot’s mobility. This test method addresses aforemen-
mine the applicability of regulatory limitations prior to use.
tioned issues of mobility.
1.9 This international standard was developed in accor-
dance with internationally recognized principles on standard- NOTE 1—Larger-sized aggregate might not be as easily packed into
robotic locomotion subsystems but might present different types of
ization established in the Decision on Principles for the
mobility challenges such as angular, rough, sharp, or broken aggregate
Development of International Standards, Guides and Recom-
pieces interfering with wheels, tracks, or other types of locomotion
mendations issued by the World Trade Organization Technical
mechanisms. These issues are out of the scope of this test method.
Barriers to Trade (TBT) Committee.
5.1.1 Sand based terrains are non-rigid in nature and could
cause a robot to turn-in-place or dig-in when the robot is
2. Referenced Documents
negotiating a tight turn. Certain robotic locomotion mecha-
2.1 ASTM Standards:
nismsmightbedesignedforothermobilitypurposesandmight
C144 Specification for Aggregate for Masonry Mortar
not create sufficient traction against the specified sand terrain.
D5821 Test Method for Determining the Percentage of
As such, extensive testing within this type of terrain may
Fractured Particles in Coarse Aggregate
expose robot design or reliability issues and lead to field
E2521 Terminology for Evaluating Response Robot Capa-
maintenance or repair.
bilities
5.1.2 The sand traverse capabilities could be affected by
E2592 Practice for Evaluating Response Robot Capabilities:
additional factors such as the weight and its distribution,
Logistics: Packaging for Urban Search and Rescue Task
ground contact areas, and control schemes for the robot. As
Force Equipment Caches
such, extensive testing within this type of terrain may also lead
E2803 Test Method for Evaluating Emergency Response
to innovations in robot design.
Robot Capabilities: Mobility: Confined Area Obstacles:
Inclined Planes
5.2 Key features of response robots are that they are
remotely operated from safe standoff distances, deployable at
3. Terminology
operational tempos, capable of operating in complex
3.1 Definitions: environments, sufficiently hardened against harsh
3.1.1 The following terms are used in this test method and
environments, reliable and field serviceable, durable or cost-
are defined in Terminology E2521: abstain, administrator or effectively disposable, and equipped with operational safe-
testadministrator,emergencyresponserobotorresponserobot,
guards.As such, a major advantage of using robots in response
fault condition, operator, operator station, remote control, operations is to enhance the safety and effectiveness of
repetition, robot, teleoperation, test event or event, test form,
responders or soldiers.
test sponsor, test suite, testing target or target, testing task or
5.3 This test method aligns user expectations with actual
task, and trial or test trial.
capabilities to understand the inherent capability trade-offs in
3.1.2 The following terms are used in this test method and
deployable systems at any given cost. For example, a design
are defined in ALFUS Framework Volume I: autonomous,
issue of the number of batteries to be packed on a robot could
autonomy,levelofautonomy,operatorcontrolunit(OCU),and
affect desired weight, endurance, or cost.Appropriate levels of
semi-autonomous.
understanding can help ensure that requirement specifications
are articulated within the limit of current capabilities.
For referenced ASTM standards, visit the ASTM website, www.astm.org, or
5.4 This test method provides a tangible representation of
contact ASTM Customer Service at service@astm.org. For Annual Book of ASTM
essential robot capabilities with quantifiable measures of per-
Standards volume information, refer to the standard’s Document Summary page on
the ASTM website.
formance. When considered with other related test methods in
NIST Special Publication 1011-I-2.0 Autonomy Levels for Unmanned Systems
the suite, it facilitates communication among communities of
(ALFUS) Framework Volume I: Terminology, Version 2.0. Available from National
robotusersandmanufacturers.Assuch,thistestmethodcanbe
Institute of Standards and Technology (NIST), 100 Bureau Dr., Stop 1070,
Gaithersburg, MD 20899-1070, http://www.nist.gov. used to:
E2992/E2992M − 17
5.4.1 Inspire technical innovation and guide manufacturers 6. Apparatus
towardimplementingcombinationsofcapabilitiesnecessaryto
6.1 The test apparatus is a fixed-size sand terrain through
perform essential mission tasks.
which the robot has to traverse (see Figs. 1 and 2 for
5.4.2 Measure and compare essential robot capabilities.
illustrations).
This test method can establish the reliability of the system to
6.1.1 The terrain size may be scaled to provide for various
perform specified tasks, highlight break-through capabilities,
levels of mobility constraints for robots under test depending
and encourage hardening of developmental systems.
on testing requirements. Three typical terrain sizes are nomi-
5.4.3 Inform purchasing decisions, conduct acceptance
nally 3.6 m [12 ft] long by 1.2 m [4 ft] wide, 7.2 m [24 ft] long
testing, and align deployment objectives with statistically
by 2.4 m [8 ft] wide, and 14.4 m [48 ft] long by 4.8 m [16 ft]
significant robot capabilities data captured through repeated
wide, respectively.Atest sponsor is authorized to specify other
testing and comparison of quantitative results.
test apparatus sizes to suit particular deployment requirements.
5.4.4 Focus operator training and measure proficiency as a
repeatable practice task that exercises actuators, sensors, and NOTE 2—Test Method E2803 and this test apparatus can be combined
to form a sand test terrain on an incline.
operator interfaces. The test method can be embedded into
training scenarios to capture and compare quantitative scores
6.1.2 Each of the terrains is fully covered with packed
even within uncontrolled environmental variables. This can
Number 8 aggregate (with nominal size of 2.36 to 9.5 mm, or
help develop, maintain, measure, and track very perishable
0.09 to 0.37 in.), as specified in Specification C144, for a
skillsovertimeandenablecomparisonsacrosssquads,regions,
minimal depth of 15 cm [6 in.], and evenly spread throughout
or national averages.
the apparatus floor. The sand shall be dry, both visibly and to
5.4.5 Although this test method was developed for response
the touch within the entire apparatus, including from the
robots, it may be applicable to other domains. Different user
surface through the bottom.
communities can set their own thresholds of acceptable per-
6.1.3 Each terrain is recommended to be fully surrounded
formance within the test method for various mission require-
by walls that are nominally 1.2 m [4 ft] tall and are typically
ments.
built out of plywood or oriented strand boards (OSB). Lumber
with a nominal cross-section of 5 by 25 cm [2 by 10 in.] could
5.5 It is recommended that users of this test method con-
alsobeusedtosurroundtheterrainandhelpfurthercontainthe
sider their particular robot requirements when interpreting the
sand.
test results. The capability evaluated in this test method alone
6.1.4 The ends of each of the terrains have end zones that
shall be interpreted according to the scope of this test method
are half as long as the width of the terrain and feature walls
and shall not be considered as an overall indication of the
completely painted with alternating black-and-white, vertical,
capability of the robot’s mobility subsystem nor of the entire
and nominally 30 cm [12 in.] wide stripes.
roboticsystem.Asingletestmethodonlycapturesthespecified
single aspect of a robot’s capabilities. A more complete 6.1.5 Fourpylonsdefineafigure-8traversepath(seeFig.2).
characterization of a robot’s capabilities requires test results The two inner pylons are placed at a distance from each other
from a wider set of test methods. equal to the width of the terrain and centered between the end
FIG. 1 Confined Area Terrains: Sand; Three Sizes
E2992/E2992M − 17
FIG. 2 Confined Area Terrains: Sand; Figure-8 Path
zones and between the sidewalls. The two outer pylons are nally 150 to 300 lux or outdoors (in the daylight), typically
placed at the edge of each end zone and centered between the measured at up to 1000 lux, nominally. The dark test condition
sidewalls. Pylons that are typically used for general traffic is specified as nominally 0.1 lux. The dark test condition is not
guidance and have nominal heights of at least 30 cm [1 ft] and specified to be lower than 0.1 lux because of the implementa-
nominalcross-sectiondiametersofatmost30cm[1ft]shallbe tion cost concerns. 0.1 lux is dark enough to require the robot
acceptable for use in this test method. to have onboard illumination—a tested robot capability—in
6.1.6 Agate that opens on a sidewall or an end zone is also order to perform the task. However, it is recognized that actual
implemented for general purpose access to the apparatus. operational environments may be darker.
6.2 In terms of lighting, the lit test condition is specified as
NOTE 3—When this test apparatus is implemented in an International
indoors (with furnished lighting), typically measured at nomi- Organization for Standardization (ISO) freight container, the 0.1 lux
E2992/E2992M − 17
darkness could be achieved by turning off all the inside lighting sources
graphs of all of the above as well as videos of routine
and covering the entrance of the container entirely with light-blocking
maintenance tasks (for example, battery change). The system
drapes.
configuration shall remain the same for all relevant tests to
6.3 Appropriate measuring devices shall be used to measure
enable direct comparison of performance and to identify
the robot’s performance within an apparatus.
capability trade-offs between different configurations. Any
number of identified system configurations can be subjected to
NOTE 4—A stopwatch may be used to measure the time and an
testing.
electronic beam-breaking sensor could be used to count the number of
repetitions. Video captured with four simultaneous feeds is highly
8.3 Determine the number of required repetitions if measur-
recommended for recording the tests. The four simultaneous video feeds
ing system capabilities.
are: an overall view of the apparatus, a detailed view of the robot
performing the task, a view of the OCU’s display showing the view from 8.3.1 Test trials shall produce enough successful repetitions
therobot’sonboardcamera,andaviewoftheoperator’shandscontrolling
to demonstrate the reliability of the system or operator neces-
therobot.Thistypeofrecordhelpsverifytestresultsandenhanceoperator
sary for the envisioned missions. The higher the ratio of
training.
successful repetitions to faults, the more reliable the system or
operator. The more repetitions completed with that ratio, the
7. Hazards
more confidence may be placed in that reliability. The calcu-
7.1 In addition to 1.8, users of this test method shall also
lated reliability and confidence levels can be determined from
address equipment preservation and human-robot coexistence
statistical tables. Some missions may require higher reliability.
concerns. Safety setups such as belays and containment walls
A test trial of 30 repetitions or more is recommended to
shall be used when there are such concerns. Environmental
establish a system’s capability. Operator proficiency trials are
conditions, such as high or low temperatures, excessive
typically time limited as specified in 8.4.
moisture, and rough terrains can be stressful, exceed the
8.3.2 A reasonable starting threshold may be at least 80 %
respective ranges within which the robot is built to properly
reliability with 80 % confidence. This can be achieved by
operate, or damage robotic components. These conditions can
performing 30 repetitions with 27 or more successes. When 30
also cause unexpected robot motions that, in turn, can have
repetitions is not feasible, this reliability and confidence may
negative effects on the humans that are nearby or on the robot
stillbeachievedifthefirsttesttrialincludes20repetitionswith
itself.
19 or more successes or 10 successful repetitions.
7.2 Identifyalltheemergencystop(E-Stop)button(s)onthe
8.3.3 Multiple trials are allowed to improve the perfor-
robot chassis and the OCU before operating or interacting with
manceofthesametestedrobotconfiguration.Ifthisisthecase,
the robot.
the latest 30 consecutive repetitions from across the multiple
trials shall be considered together when determining the ratio
7.3 While the robot is activ
...