ASTM E2802/E2802M-21e1

This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the
Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
´1
Designation: E2802/E2802M − 21
Standard Test Method for
Evaluating Response Robot Mobility Using Variable Hurdle
Obstacles
ThisstandardisissuedunderthefixeddesignationE2802/E2802M;thenumberimmediatelyfollowingthedesignationindicatestheyear
of original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval.
A superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
ε NOTE—Editorial corrections were made to Table 1 in May 2021.
INTRODUCTION
The robotics community needs ways to measure whether a particular robot is capable of performing
specificmissionsincomplex,unstructured,andoftenhazardousenvironments.Thesemissionsrequire
various combinations of elemental robot capabilities. Each capability can be represented as a test
method with an associated apparatus to provide tangible challenges for various mission requirements
and performance metrics to communicate results. These test methods can then be combined and
sequenced to evaluate essential robot capabilities and remote operator proficiencies necessary to
successfully perform intended missions.
TheASTM International Standards Committee on Homeland SecurityApplications (E54) specifies
these standard test methods to facilitate comparisons across different testing locations and dates for
diverse robot sizes and configurations. These standards support robot researchers, manufacturers, and
user organizations in different ways. Researchers use the standards to understand mission
requirements, encourage innovation, and demonstrate break-through capabilities. Manufacturers use
the standards to evaluate design decisions, integrate emerging technologies, and harden systems.
Emergency responders and soldiers use them to guide purchasing decisions, align deployment
expectations, and focus training with standard measures of operator proficiency. Associated usage
guides describe how these standards can be applied to support various objectives.
Several suites of standards address these elemental capabilities including maneuvering, mobility,
dexterity, sensing, energy, communications, durability, proficiency, autonomy, and logistics. This
standard is part of the Mobility suite of test methods.
1. Scope used to evaluate assistive or autonomous behaviors intended to
improve the effectiveness or efficiency of remotely operated
1.1 This test method is intended for remotely operated
systems.
ground robots operating in complex, unstructured, and often
1.3 Different user communities can set their own thresholds
hazardous environments. It specifies the apparatuses,
of acceptable performance within this test method for various
procedures, and performance metrics necessary to measure the
mission requirements.
capability of a robot to negotiate an obstacle in the form of
hurdles.Thistestmethodisoneofseveralrelatedmobilitytests
1.4 Performing Location—This test method may be per-
that can be used to evaluate overall system capabilities.
formed anywhere the specified apparatuses and environmental
conditions can be implemented.
1.2 Theroboticsystemincludesaremoteoperatorincontrol
of most functionality, so an onboard camera and remote
1.5 Units—The International System of Units (a.k.a. SI
operator display are typically required.This test method can be
Units) and U.S. Customary Units (a.k.a. Imperial Units) are
used throughout this document. They are not mathematical
conversions. Rather, they are approximate equivalents in each
system of units to enable use of readily available materials in
This test method is under the jurisdiction of ASTM Committee E54 on
Homeland Security Applications and is the direct responsibility of Subcommittee
different countries. The differences between the stated dimen-
E54.09 on Response Robots.
sions in each system of units are insignificant for the purposes
Current edition approved March 1, 2021. Published March 2021. Originally
of comparing test method results, so each system of units is
approved in 2011. Last previous edition approved in 2020 as E2802 – 11 (2020).
DOI: 10.1520/E2802_E2802M-21E01. separately considered standard within this test method.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
´1
E2802/E2802M − 21
1.6 This standard does not purport to address all of the 3. Terminology
safety concerns, if any, associated with its use. It is the
3.1 Definitions—The following terms are used in this test
responsibility of the user of this standard to establish appro-
method and are defined in Terminology E2521: abstain,
priate safety, health, and environmental practices and deter-
administrator or test administrator, emergency response robot
mine the applicability of regulatory limitations prior to use.
or response robot, fault condition, operator, operator station,
1.7 This international standard was developed in accor-
remote control, repetition, robot, teleoperation, test event or
dance with internationally recognized principles on standard-
event, test form, test sponsor, test suite, testing target or target,
ization established in the Decision on Principles for the
testing task or task, and trial or test trial.
Development of International Standards, Guides and Recom-
mendations issued by the World Trade Organization Technical
3.2 The following terms are used in this test method and are
Barriers to Trade (TBT) Committee.
defined in ALFUS Framework Volume I:3: autonomous,
autonomy, level of autonomy, operator control unit (OCU),and
2. Referenced Documents
semi-autonomous.
2.1 ASTM Standards:
3.3 Definitions of Terms Specific to This Standard:
E2521 Terminology for Evaluating Response Robot Capa-
3.3.1 pallet, n—a stackable unit with an Oriented Strand
bilities
Board (OSB) top surface or similar material sized to fit inside
E2592 Practice for Evaluating Response Robot Capabilities:
a subfloor.
Logistics: Packaging for Urban Search and Rescue Task
Force Equipment Caches 3.3.2 subfloor, n—an underlayment of OSB or similar ma-
2.2 Other Standards: terial with dimensional lumber borders used to affix multiple
subfloors to one another and can contain apparatus elements
NIST Special Publication 1011-I-2.0 Autonomy Levels for
such as terrains or obstacles.
Unmanned Systems (ALFUS) Framework, Volume 1:
Terminology, Version 2.04
NIST Special Publication 1011-II-1.0 Autonomy Levels for 4. Summary of Test Method
Unmanned Systems (ALFUS) Framework, Volume I:3
4.1 This test method is performed by a remote operator that
cannotseeorheartherobotwithinthetestapparatus.Therobot
traversesthroughadefinedareatonegotiatethevariablehurdle
For referenced ASTM standards, visit the ASTM website, www.astm.org, or
obstacle with or without walls for confinement (see Fig. 1).
contact ASTM Customer Service at service@astm.org. For Annual Book of ASTM
This test method requires the robot to overcome challenges
Standards volume information, refer to the standard’s Document Summary page on
the ASTM website.
such as pitch, roll, traction, and control of variable chassis
Available from National Institute of Standards and Technology (NIST), 100
shape and articulators.
Bureau Dr., Stop 1070, Gaithersburg, MD 20899-1070, http://www.nist.gov.
FIG. 1 (A) View of the Variable Hurdle Obstacle
Shown Heights: 20 cm [8 in.], 30 cm [12 in.], 40 cm [16 in.]
´1
E2802/E2802M − 21
FIG. 1 (B) View of a Test Apparatus showing the Open, Rectangular Confinement,
and Square Confinement Approach/Exit Areas; and Example Robot Traversal Paths (continued)
4.2 The robot traverses a path as shown in Fig. 1. The robot unobstructed areas. The rectangular confinement and square
starts on theA-side of the apparatus, crosses to the B-side into confinement configurations use walls around the approach/exit
the nearest approach area, traverses over the variable hurdle areas. The walls are used to define the robot’s path and are
obstacle to the exit area on the opposite end of the apparatus, representative of operating in a confined environment. The
and then crosses to the A-side to complete each repetition. All square configuration has half of the available area as the
repetitions alternate directions through the apparatus. rectangular configuration.
4.3 The robot traverses the path in one of two operationally- 4.5 Potential Faults Include:
relevant driving orientations: unrestricted or forward/reverse. 4.5.1 Any contact by the robot with the apparatus that
Unrestrictedallowstherobottotraversethepathinanydriving requires adjustment or repair to return the apparatus to the
orientation throughout the test. Forward/reverse requires the initial condition.
robot to alternate between forward and reverse driving orien- 4.5.2 Any visual, audible, or physical interaction that assists
tationsforsubsequentrepetitionsthroughoutthetest.Resulting either the robot or the remote operator.
data from the two driving orientations are not comparable to
4.6 Test trials shall produce enough successful repetitions to
one another.
demonstrate the reliability of the system capability or the
4.4 There are three apparatus configurations: open, rectan- remote operator proficiency. A complete trial of 10 to 30
gular confinement, and square confinement.Inthe open repetitions should take 10 to 30 min to complete. When
configuration, no walls are used around the approach/exit measuring system capabilities, it is important to allow enough
areas. The open configuration is representative of operating in time to capture a complete trial with an expert operator. When
´1
E2802/E2802M − 21
measuring operator proficiency, it is important to limit the time multiple test methods can guide manufacturers toward imple-
of the trial so that novice and expert operators are similarly menting the combinations of capabilities necessary to perform
fatigued. essential mission tasks.
4.7 There are three metrics to consider when calculating the
6. Apparatus
results of a test trial. They should be considered in the
following order of importance: completeness score, reliability, 6.1 The equipment required to perform this test method
and efficiency.The results from open, rectangular confinement,
includes pallets, subfloors, pipes, walls (only for rectangular
and square confinement configurations are not comparable confinement and square confinement configurations), and a
because they represent different difficulties and clearances.
timer. The apparatus consists of subfloors, walls, and the
variable hurdle obstacle (see Fig. 2). The main apparatus
5. Significance and Use dimension to consider is the minimum clearance width (W) for
the robot. The minimum clearance width should be chosen to
5.1 This test method is part of an overall suite of related test
representtheintendeddeploymentenvironmentorbasedonthe
methods that provide repeatable measures of robotic system
size of the robot, or both. The minimum clearance width is
mobility and remote operator proficiency. The variable hurdle
typically set to 120 cm [4 ft], 60 cm [2 ft], or 30 cm [1 ft] to
obstacle as described challenges robotic system locomotion,
efficiently use available construction materials, although other
suspension systems to maintain traction, rollover tendencies,
Fig. 3). All apparatus
apparatus sizes can be used (see
self-righting (if necessary), chassis shape variability (if
dimensions scale proportionally with the minimum clearance
available), and remote situational awareness by the operator.
width (see Fig. 4). For example, the width of the variable
As such, the variable hurdle obstacle can be used to represent
hurdle obstacle apparatus is 1W, and the length of the variable
obstacles in the environment, such as railroad tracks, curbs,
hurdle obstacle apparatus is either 3W (square confinement
and debris.
configuration) or 5W (rectangular confinement and open con-
5.2 The scale of the apparatus can vary to provide different
figurations). Resulting data from a specific minimum clearance
constraints representative of typical obstacle spacing in the
width of the apparatus is not comparable to data from other
intended deployment environment. For example, the three
apparatuses with different minimum clearance widths.
configurations can be representative of repeatable complexity
6.2 The apparatus consists of two symmetrical approach/
for unobstructed obstacles (open configuration), relatively
exit areas on either side of a variable hurdle obstacle.There are
open parking lots with spaces between cars (rectangular
three configurations of the apparatus: open, rectangular
confinementconfiguration),orwithinbus,train,orplaneaisles,
confinement, and square confinement (see Fig. 3). The selec-
or dwellings with hallways and doorways (square confinement
tion of apparatus configuration should correspond to intended
configuration).
deployment environment. The open configuration does not use
5.3 The test apparatuses are low cost and easy to fabricate
walls in the approach areas on either side of the variable hurdle
so they can be widely replicated. The procedure is also simple
obstacle, allowing for unobstructed robot movement. The
to conduct. This eases comparisons across various testing
approach areas in the rectangular confinement configuration
locations and dates to determine best-in-class systems and
measure 2W by 1W and are bounded by walls taller than the
operators.
robot to obstruct robot movement. The approach areas in the
5.4 Evaluation—This test method can be used in a con- square confinement configuration measure 1W by 1W and are
trolled environment to measure baseline capabilities. The bounded by walls taller than the robot to further obstruct robot
variable hurdle obstacle can also be embedded into operational movement. Resulting data from a specific configuration of the
training scenarios to measure degradation due to uncontrolled apparatus is not comparable to data from other apparatuses
variables in lighting, weather, radio communications, GPS with different configurations.
accuracy, etc.
6.3 Pallet and Pipe—The variable hurdle obstacle is con-
5.5 Procurement—This test method can be used to identify structed of multiple pallets which are made of OSB or similar
material, dimensional lumber, and pipes on both ends. The
inherent capability trade-offs in systems, make informed pur-
chasing decisions, and verify performance during acceptance thickness of the pallet and the outer diameter of the pipe are
relative to the scale of the apparatus (see Table 1). The pallet
testing. This aligns requirement specifications and user expec-
tations with existing capability limits. is fabricated to fit within the rails of the subfloor. The ground
dimension of the hurdle is set to 1W and the overall height of
5.6 Training—This test method can be used to focus opera-
the variable hurdle obstacle (H) is adjustable by stacking
tortrainingasarepeatablepracticetaskorasanembeddedtask
multiple pallets and pipes (see Fig. 1 and Fig. 5).
within training scenarios. The resulting measures of remote
operator proficiency enable tracking of perishable skills over 6.4 Subfloor—The subfloor’s surface is constructed of OSB
or similar material and rails are dimensional lumber that
time, along with comparisons of performance across squads,
regions, or national averages. surround the border of the subfloor. Each subfloor is 2W by
1W. A gap in the rails halfway along the side measuring 2W
5.7 Innovation—This test method can be used to inspire
will allow containment wall to be inserted (see Fig. 6).
technical innovation, demonstrate break-through capabilities,
andmeasurethereliabilityofsystemsperformingspecifictasks 6.5 Walls to Confine the Robot Path—The walls placed
within an overall mission sequence. Combining or sequencing within the rectangular confinement and square confinement
´1
E2802/E2802M − 21
FIG. 2 View of a Test Apparatus with Labeled Components
FIG. 3 Testing Apparatus is Scalable to Represent Different Environments
configurations provide physical and visual guidance for the test. This ensures that all parts of the robot remain contained
remote robot operator to traverse the variable hurdle obstacle
within the designated area defined by the walls. The walls
(seeFig.7).Thewallscanbemadefromanysolidmaterialand
should be sturdy and easily repaired or replaced.
mustbetallerthanthehighestpointoftherobotthroughoutthe
´1
E2802/E2802M − 21
FIG. 4 Top View of a Test Apparatus showing the Dimensions and Labeled
Open, Rectangular Confinement, and Square Confinement Approach/Exit Areas
TABLE 1 Corresponding Height of the Hurdle Pallet and Pipes when used in Different Apparatus Scales
Apparatus Width (W) Nominal Thickness of the Pallet using Nominal Outer Diameter of the Pipes Pipe Stack Height (P) Tolerance
Dimensional Lumber
†
120 cm [48 in.] 10 cm [4 in.] 10 cm [4 in.] H ± 12 mm [0.5 in.]
†
60 cm [24 in.] 5cm[2in.] 5cm[2in.] H±6mm[0.25in.]
†
30 c
...