ASTM E3349/E3349M-22

This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the
Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
Designation:E3349/E3349M −22
Standard Test Method for
Evaluating Ground Robot Capabilities and Remote Operator
Proficiency: Terrains: K-Rails
ThisstandardisissuedunderthefixeddesignationE3349/E3349M;thenumberimmediatelyfollowingthedesignationindicatestheyear
of original adoption or, in the case of revision, the year of last revision.Anumber in parentheses indicates the year of last reapproval.
A superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
INTRODUCTION
Theroboticscommunityneedswaystomeasurewhetheraparticularrobotiscapableofperforming
specificmissionsincomplex,unstructured,andoftenhazardousenvironments.Thesemissionsrequire
various combinations of elemental robot capabilities. Each capability can be represented as a test
method with an associated apparatus to provide tangible challenges for various mission requirements
and performance metrics to communicate results. These test methods can then be combined and
sequenced to evaluate essential robot capabilities and remote operator proficiencies necessary to
successfully perform intended missions.
TheASTM International Standards Committee on Homeland SecurityApplications (E54) specifies
these standard test methods to facilitate comparisons across different testing locations and dates for
diverse robot sizes and configurations. These standards support robot researchers, manufacturers, and
user organizations in different ways. Researchers use the standards to understand mission
requirements, encourage innovation, and demonstrate break-through capabilities. Manufacturers use
the standards to evaluate design decisions, integrate emerging technologies, and harden systems.
Emergency responders and soldiers use them to guide purchasing decisions, align deployment
expectations, and focus training with standard measures of operator proficiency. Associated usage
guides describe how these standards can be applied to support various objectives.
Several suites of standards address these elemental capabilities including maneuvering, mobility,
dexterity, sensing, energy, communications, durability, proficiency, autonomy, and logistics. This
standard is part of the Terrain suite of test methods.
1. Scope mous behaviors that improve the effectiveness or efficiency of
the overall system are encouraged.
1.1 This test method is intended for remotely operated
ground robots operating in complex, unstructured, and often
1.3 Different user communities can set their own thresholds
hazardous environments. It specifies the apparatuses,
of acceptable performance within this test method for various
procedures, and performance metrics necessary to measure the
mission requirements.
capabilityofarobottotraversecomplexterrainsintheformof
1.4 Performing Location—This test method may be per-
k-rails. This test method is one of several related Terrain tests
formed anywhere the specified apparatuses and environmental
that can be used to evaluate overall system capabilities.
conditions can be implemented.
1.2 Theroboticsystemincludesaremoteoperatorincontrol
of all functionality, so an onboard camera and remote operator
1.5 Units—The International System of Units (a.k.a. SI
display are typically required. Assistive features or autono-
Units) and U.S. Customary Units (a.k.a. Imperial Units) are
used throughout this document. They are not mathematical
conversions. Rather, they are approximate equivalents in each
This test method is under the jurisdiction of ASTM Committee E54 on
system of units to enable use of readily available materials in
Homeland Security Applications and is the direct responsibility of Subcommittee
different countries. This avoids excessive purchasing and
E54.09 on Response Robots.
fabrication costs. The differences between the stated dimen-
Current edition approved June 1, 2022. Published July 2022. DOI: 10.1520/
E3349_E3349M-22. sions in each system of units are insignificant for the purposes
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
E3349/E3349M−22
of comparing test method results, so each system of units is point in the apparatus through which the robot should pass.
separately considered standard within this test method. Consult Section 6 for the overall measurements and dimen-
sions of the apparatus at each scale.
1.6 This standard does not purport to address all of the
safety concerns, if any, associated with its use. It is the 3.3.2 cross-over slope, n—anoptionaltestconfigurationthat
responsibility of the user of this standard to establish appro- augments the center of the terrain apparatus which positions
priate safety, health, and environmental practices and deter- two portions of terrain on adjacent slopes of 15° in opposite
mine the applicability of regulatory limitations prior to use. directions.
1.7 This international standard was developed in accor-
3.3.3 diagonal rail, n—a solid piece of dimensional lumber
dance with internationally recognized principles on standard-
that is sized to fit horizontally inside a subfloor at a 45° angle
ization established in the Decision on Principles for the
to the direction of travel.
Development of International Standards, Guides and Recom-
3.3.4 subfloor, n—an underlayment of Oriented Strand
mendations issued by the World Trade Organization Technical
Board (OSB) or similar material with dimensional lumber
Barriers to Trade (TBT) Committee.
borders used to affix multiple subfloors to one another and can
contain apparatus elements such as terrains or obstacles.
2. Referenced Documents
2.1 ASTM Standards:
4. Summary of Test Method
E2521Terminology for Evaluating Response Robot Capa-
4.1 This test method is performed by a remote operator, out
bilities
of sight and sound of the robot, while controlling the robot
2.2 Other Standards:
within the test apparatus.The robot follows one of two defined
NISTSpecial Publication 1011–I–2.0 Autonomy Levels for
paths in the specified terrain requiring the robot to overcome
Unmanned Systems (ALFUS) Framework, Volume 1:
challenges including pitch, roll, traction, and control of vari-
Terminology, Version 2.04
able chassis shape and articulators within open or confined
spaces.
3. Terminology
4.2 Thefigure-8path(forward)isacontinuousforwardpath
3.1 Definitions—The following terms are used in this test
throughtheterrainwithalternatingleftandrightturnstoavoid
method and are defined in Terminology E2521: abstain,
barriers. It can be used to demonstrate terrain traversal over
administrator or test administrator, emergency response robot
long distances within a relatively small apparatus. The con-
or response robot, fault condition, operator, operator station,
tinuous traverse is shown as the white path (see Figs. 1 and 2).
remote control, repetition, robot, teleoperation, test event or
event, test form, test sponsor, test suite, testing target or target,
4.3 Thezig-zagpath(forward/reverse)isanend-to-endpath
testing task or task, and trial or test trial.
that requires forward and reverse traversal through the terrain
with alternating left and right turns to avoid barriers. This can
3.2 Thefollowingtermsareusedinthistestmethodandare
be used to demonstrate traversal of the terrain within confined
defined in ALFUS Framework Volume I:3: autonomous,
spaces. The down-range traverse, shown as the white path, is
autonomy, level of autonomy, operator control unit (OCU),and
performed in a forward orientation and the up-range traverse,
semi-autonomous.
shownastheblackpath,isperformedinreverse(seeFig.1and
3.3 Definitions of Terms Specific to This Standard:
Fig. 3).
3.3.1 apparatus clearance width (W), n—a specification for
4.4 The robot starts on one side or the other of a lane full of
the apparatus dimensions chosen from one of four possible
fabricated k-rail terrain at a chosen scale. The robot follows
measurements, based on the intended robot deployment envi-
either the figure-8 path (forward) or the zig-zag path (forward/
ronment:
reverse) between the two barriers. The figure-8 path (forward)
240 cm 6 2.5 cm tolerance [96 in. 6 1 in. tolerance], such
repetition is completed when the robot crosses the start/end
as open and outdoor public spaces;
centerline of the lane without a fault after approximately
120 cm 6 2.5 cm tolerance [48 in. 6 1 in. tolerance], such
following the white path. The zig-zag path (forward/reverse)
as indoor spaces in accessibility-compliant buildings;
repetition is completed when the robot crosses the start/end
60 cm 6 1.3 cm tolerance [24 in. 6 0.5 in. tolerance],
centerline without a fault after approximately following the
residences and aisles of public transportation;
white and black paths.
30 cm 6 1.3 cm tolerance [12 in. 6 0.5 in. tolerance],
cluttered indoor spaces, ductwork, and voids in collapsed
4.5 Potential Faults Include:
structures.
4.5.1 Any contact by the robot with the apparatus that
3.3.1.1 Discussion—The measures for these scales are
requires adjustment or repair to return the apparatus to the
nominalanddonotrepresentthemeasurementofthenarrowest
initial condition;
4.5.2 Anyvisual,audible,orphysicalinteractionthatassists
either the robot or the remote operator; and
For referenced ASTM standards, visit the ASTM website, www.astm.org, or
4.5.3 Leaving the apparatus during the trial.
contact ASTM Customer Service at service@astm.org. For Annual Book of ASTM
Standards volume information, refer to the standard’s Document Summary page on
4.6 By default, the terrain apparatus is fabricated such that
the ASTM website.
it is not inclined and sits flat on the ground. An optional
Available from National Institute of Standards and Technology (NIST), 100
Bureau Dr., Stop 1070, Gaithersburg, MD 20899-1070, http://www.nist.gov. configuration of this test method is available wherein a
E3349/E3349M−22
FIG. 1Overview of the K-rail Terrain Apparatus
FIG. 2Top View showing the Figure-8 Path (forward) Defined by the Barriers
cross-over slope element is added to the center area of the repetitions in either one of the defined paths should take 10 to
terrain which positions two portions of terrain on adjacent 30 min to complete. When measuring system capabilities, it is
slopes of 15° in opposite directions (see Fig. 4). When this importanttoallowenoughtimetocaptureacompletetrialwith
configuration is used, only the zig-zag path can be used. an expert operator. When measuring operator proficiency, it is
importanttolimitthetimeofthetrialsothatnoviceandexpert
4.7 Testtrialsshallproduceenoughsuccessfulrepetitionsto
operators are similarly fatigued.
demonstrate the reliability of the system capability or the
remote operator proficiency to the desired level of statistical 4.8 There are three metrics to consider when calculating the
significance (see Section 9). A complete trial of 10 to 30 results of a test trial. They should be considered in the
E3349/E3349M−22
FIG. 3Top View showing the Zig-Zag Path (forward/reverse) Defined by the Barriers
FIG. 4The K-rail Terrain Apparatus in the Cross-over Slope Configuration
following order of importance: completeness score, reliability, test apparatus are also not comparable because they represent
andefficiency.Theresultsfromthefigure-8path(forward)and different clearances and distances (Fig. 5).
the zig-zag path (forward/reverse) are not comparable because
5. Significance and Use
they measure different capabilities. Similarly, the results from
the cross-over slope apparatus configuration are not compa- 5.1 Thistestmethodispartofanoverallsuiteofrelatedtest
rable to results from the apparatus in its default flat configu- methods that provide repeatable measures of robotic system
rationwiththezig-zagpath.Theresultsfromdifferentscalesof mobility and remote operator proficiency. This k-rail terrain
E3349/E3349M−22
FIG. 5Both Paths are Scalable to Represent Different Environments
specifically challenges robotic system locomotion, suspension multiple test methods can guide manufacturers toward imple-
systems to maintain traction, rollover tendencies, self-righting menting the combinations of capabilities necessary to perform
in complex terrain (if necessary), chassis shape variability (if essential mission tasks.
available), and remote situational awareness by the operator.
6. Apparatus
As such, it can be used to represent modest to challenging
(when the cross-over slope configuration is used) outdoor
6.1 The apparatus consists of diagonal rails set into sub-
terrain complexity or indoor debris within confined areas.
floors to form the terrain, barriers to define the robot path, an
optional containment structure, and an optional set of cross-
5.2 The overall size of the terrain apparatus can vary to
overslopestands.Themainapparatusdimensiontoconsideris
provide different constraints depending on the typical obstacle
the apparatus clearance width (W) for the robot, which can be
spacingoftheintendeddeploymentenvironment.Forexample,
set to 240 cm [96 in.] with 62.5 cm [1 in.] tolerance, 120cm
the terrain with containment walls can be sized to represent
[48 in.] with 62.5 cm [1 in.] tolerance, 60 cm [24 in.] with
repeatable complexity within bus, train, or plane aisles; dwell-
61.3 cm [0.5 in.] tolerance, or 30 cm [12 in.] with 61.3 cm
ings with hallways and doorways; relatively open parking lots
[0.5 in.] tolerance. The dimension chosen for W should
with spaces between cars; or unobstructed terrains.
representtheintendeddeploymentenvironmentorbebasedon
5.3 The test apparatuses are low cost and easy to fabricate
thesizeoftherobot(thatis,therobotshallbeabletomaneuver
so they can be widely replicated. The procedure is also simple
within the selected dimensions of the apparatus), or both. All
to conduct. This eases comparisons across various testing
apparatus dimensions scale proportionally with W; the overall
locations and dates to determine best-in-class systems and
width of the terrain lane is 2W, the overall length of the terrain
operators.
lane is at least 6W, and the length of the barriers is 1W. The
5.4 Evaluation—This test method can be used in a con-
diagonal rail height (H) also scales with W; see Table 1 and
trolled environment to measure baseline capabilities. It can
Fig.6.Theoveralllengthoftheterrainlanecanbemadelonger
also be embedded into operational training scenarios to mea-
to accommodate larger robots that need more space to maneu-
sure degradation due to uncontrolled variables in lighting,
ver around the barriers while staying on the terrain. When
weather, radio communications, GPS accuracy, etc.
choosing a specific apparatus clearance width, note that the
resulting data is not comparable to other apparatuses with
5.5 Procurement—This test method can be used to identify
different clearance widths.
inherent capability trade-offs in systems, make informed pur-
chasing decisions, and verify performance during acceptance
6.2 K-rail Terrain Panel—The k-rail terrain panel consists
testing. This aligns requirement specifications and user expec-
of a subfloor and two diagonal rails. Each subfloor is 2W by
tations with existing capability limits.
1W. The subfloor’s surface is constructed of OSB or similar
material with dimensional lumber along the edges measuring
5.6 Training—This test method can be used to focus opera-
tortrainingasarepeatablepracticetaskorasanembeddedtask
within training scenarios. The resulting measures of remote
TABLE 1 Corresponding Height of the Diagonal Rail when Used
operator proficiency enable tracking of perishable skills over
in Different Apparatus Clearance Widths
time, along with comparisons of performance across squads,
Apparatus Clearance Width (W) Nominal Height (H) of the Diagonal Rail
regions, or national averages.
using Dimensional Lumber
5.7 Innovation—This test method can be used to inspire 240 cm [96 in.] 20 cm [8 in.]
120 cm [48 in.] 10 cm [4 in.]
technical innovation, demonstrate break-through capabilities,
60 cm [24 in.] 5 cm [2 in.]
andmeasurethereliabilityofsystemsperformingspecifictasks
30 cm [12 in.] 2.5 cm [1 in.]
within an overall mission sequence. Combining or sequencing
E3349/E3349M−22
FIG. 6Top View of a Test Apparatus showing Dimensions Scale Proportionally to the Apparatus Clearance Width (W)
2W. On top of each subfloor are two diagonal rails made of be lined with wood panels to cover the corrugated steel and
dimensional lumber which fit into the subfloor at 45°. The haveenoughthicknesstofillanygapsbetweenthewallandthe
height of the diagonal rail (H) scales with the apparatus terrain. See Fig. 12.
clearance width (W); see Table 1. Two additional blocks of
6.6 Other Devices—A timer is used to measure the elapsed
dimensional lumber are attached to the inside of the subfloor
time of the trial. It provides a deterministic indication of trial
and hold the diagonal rails in place. When two k-rail terrain
start and end times to minimize uncertainty. It can count-up or
panels are adjacent to one another along the longest edge, they
count-down but should have a settable duration in minutes. A
must be oriented opposite of each other such that the diagonal
stopwatchcanalsobeused.Alightmeterisnecessarytoensure
rails form a diamond shape. Additional dimensional lumber
the environment is considered lighted (>150lx) or dark
can be used on the outside of the apparatus to connect two
(<0.1lx).Athermometer is necessary to meas
...