ASTM E3310/E3310M-22

This document is not an ASTM standard and is intended only to provide the user of an ASTM standard an indication of what changes have been made to the previous version. Because
it may not be technically possible to adequately depict all changes accurately, ASTM recommends that users consult prior editions as appropriate. In all cases only the current version
of the standard as published by ASTM is to be considered the official document.
Designation: E3310/E3310M − 21 E3310/E3310M − 22
Standard Test Method for
Evaluating Response Robot Mobility Across Ground Robot
Capabilities and Remote Operator Proficiency:
Maneuvering: Align Ground Contacts with Parallel Rails
This standard is issued under the fixed designation E3310/E3310M; the number immediately following the designation indicates the year
of original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval.
A superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
INTRODUCTION
The robotics community needs ways to measure whether a particular robot is capable of performing
specific missions in complex, unstructured, and often hazardous, environments. These missions
require various combinations of elemental robot capabilities. Each capability can be represented as a
test method with an associated apparatus to provide tangible challenges for various mission
requirements and performance metrics to communicate results. These test methods can then be
combined and sequenced to evaluate essential robot capabilities and remote operator proficiencies
necessary to successfully perform intended missions.
The ASTM International Standards Committee on Homeland Security Applications (E54) specifies
these standard test methods to facilitate comparisons across different testing locations and dates for
diverse robot sizes and configurations. These standards support robot researchers, manufacturers, and
user organizations in different ways. Researchers use the standards to understand mission
requirements, encourage innovation, and demonstrate break-through capabilities. Manufacturers use
the standards to evaluate design decisions, integrate emerging technologies, and harden systems.
Emergency responders and soldiers use them to guide purchasing decisions, align deployment
expectations, and focus training with standard measures of operator proficiency. Associated usage
guides describe how these standards can be applied to support various objectives.
Several suites of standards address these elemental capabilities including maneuvering, mobility,
dexterity, sensing, energy, communications, durability, proficiency, autonomy, and logistics. This
standard is part of the MobilityManeuvering suite of test methods.
1. Scope
1.1 This test method is intended for remotely operated ground robots operating in complex, unstructured, and often hazardous
environments. It specifies the apparatuses, procedures, and performance metrics necessary to measure the capability of a robot to
maneuver align its ground contacts while maneuvering across parallel rails. This test method is one of several related
mobilitymaneuvering tests that can be used to evaluate overall system capabilities.
1.2 The robotic system includes a remote operator in control of most functionality, so an onboard camera and remote operator
display are typically required. This test method can be used to evaluate assistive or autonomous behaviors intended to improve the
effectiveness or efficiency of remotely operated systems.
This test method is under the jurisdiction of ASTM Committee E54 on Homeland Security Applications and is the direct responsibility of Subcommittee E54.09 on
Response Robots.
Current edition approved Nov. 15, 2021May 1, 2022. Published December 2021May 2022. Originally approved in 2021. Last previous edition approved in 2021 as
E3310/E3310M – 21. DOI: 10.1520/E3310_E3310M-21.10.1520/E3310_E3310M-22.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
E3310/E3310M − 22
1.3 Different user communities can set their own thresholds of acceptable performance within this test method for various mission
requirements.
1.4 Performing Location—This test method may be performed anywhere the specified apparatuses and environmental conditions
can be implemented.
1.5 Units—The International System of Units (a.k.a. SI Units) and U.S. Customary Units (a.k.a. Imperial Units) are used
throughout this document. They are not mathematical conversions. Rather, they are approximate equivalents in each system of
units to enable use of readily available materials in different countries. The differences between the stated dimensions in each
system of units are insignificant for the purposes of comparing test method results, so each system of units is separately considered
standard within this test method.
1.6 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility
of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of
regulatory limitations prior to use.
1.7 This international standard was developed in accordance with internationally recognized principles on standardization
established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued
by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
2. Referenced Documents
2.1 ASTM Standards:
E2521 Terminology for Evaluating Response Robot Capabilities
E2592 Practice for Evaluating Response Robot Capabilities: Logistics: Packaging for Urban Search and Rescue Task Force
Equipment Caches
2.2 Other Standard:
NIST Special Publication 1011–I–2.0 Autonomy Levels for Unmanned Systems (ALFUS) Framework, Volume 1: Terminology,
Version 2.04
3. Terminology
3.1 Definitions—The following terms are used in this test method and are defined in Terminology E2521: abstain, administrator
or test administrator, emergency response robot or response robot, fault condition, operator, operator station, remote control,
repetition, robot, teleoperation, test event or event, test form, test sponsor, test suite, testing target or target, testing task or task,
and trial or test trial.
3.2 The following terms are used in this test method and are defined in ALFUS Framework Volume I:3: autonomous, autonomy,
level of autonomy, operator control unit (OCU), and semi-autonomous.
3.3 Definitions of Terms Specific to This Standard:
3.3.1 apparatus clearance width (W),n—a specification for the apparatus dimensions chosen from one of four possible
measurements, based on the intended robot deployment environment:
240 cm 6 2.5 cm tolerance [96 in. 6 1 in. tolerance], such as open and outdoor public spaces;
120 cm 6 2.5 cm tolerance [48 in. 6 1 in. tolerance], such as indoor spaces in accessibility-compliant buildings;
60 cm 6 1.3 cm tolerance [24 in. 6 0.5 in. tolerance], residences and aisles of public transportation; or
30 cm 6 1.3 cm tolerance [12 in. 6 0.5 in. tolerance], cluttered indoor spaces, ductwork, and voids in collapsed structures.
3.3.1.1 Discussion—
The measures for these scales are nominal and do not represent the measurement of the narrowest point in the apparatus through
which the robot should pass. Consult the Section 6 (Apparatus) for the overall measurements and dimensions of the apparatus at
each scale.
3.3.2 pallet, n—a stackable unit with an Oriented Strand Board (OSB) top surface or similar material sized to fit inside a subfloor.
For referenced ASTM standards, visit the ASTM website, www.astm.org, or contact ASTM Customer Service at service@astm.org. For Annual Book of ASTM Standards
volume information, refer to the standard’s Document Summary page on the ASTM website.
Available from National Institute of Standards and Technology (NIST), 100 Bureau Dr., Stop 1070, Gaithersburg, MD 20899-1070, http://www.nist.gov.
E3310/E3310M − 22
3.3.3 parallel rails, n—two solid pieces of dimensional lumber positioned parallel to each other with variable distance between
them.
3.3.4 subfloor, n—an underlayment of OSB or similar material with dimensional lumber borders used to affix multiple subfloors
to one another and can contain apparatus elements such as terrains or obstacles.
4. Summary of Test Method
4.1 This test method is performed by a remote operator that cannot see or hear the operator, out of sight and sound of the robot,
while controlling the robot within the test apparatus. The robot traverses through a defined area to maneuver across the parallel
rails with or without walls for confinement (see Fig. 1). The separation distance between the two parallel rails is set based on the
width of the robot’s ground contacts to produce a narrow traversal area. This test method requires the robot to overcome challenges
such as controlled movements, effective camera positioning, control of variable chassis shape and articulators, and remote
situational awareness by the operator.
4.2 The robot traverses a path as shown in Fig. 1. The robot starts on the A-side of the apparatus, crosses to the B-side into the
nearest approach area, maneuvers across the parallel rails without making contact with the ground surface to the exit area on the
opposite end of the apparatus, and then crosses to the A-side to complete each repetition. All repetitions alternate directions through
the apparatus.
4.3 The robot traverses the path in one of two operationally-relevant driving orientations: unrestricted or forward/
reverse.Unrestricted allows the robot to traverse the path in any driving orientation throughout the test. Forward/reverse requires
the robot to alternate, for each repetition, driving in forward and reverse. As repetitions also alternate directions through the
apparatus, this means that the robot shall not rotate between ending one repetition and starting the next. Resulting data from the
two driving orientations are not comparable to one another.
4.4 There are three apparatus configurations: open,rectangular confinement, and square confinement. In the open configuration,
no walls are used around the approach/exit areas. The open configuration is representative of operating in unobstructed areas. The
rectangular confinement and square confinement configurations use walls around the approach/exit areas. The walls are used to
define the robot’s path and are representative of operating in a confined environment. The square configuration has half of the
available area as the rectangular configuration.
4.5 Potential Faults Include:
4.5.1 Any contact by the robot with the ground surface in the parallel rails area;
FIG. 1 (A) View of the Parallel Rails; Shown Separations: 30 cm [12 in.], 15 cm [6 in.]
E3310/E3310M − 22
FIG. 1 (B) View of the Parallel Rails Apparatus showing the Open, Rectangular, and Square Confinement Approach/exit Areas
and Example Robot Traversal Paths (continued)
4.5.2 Any contact by the robot with the apparatus that requires adjustment or repair to return the apparatus to the initial condition;
and
4.5.3 Any visual, audible, or physical interaction that assists either the robot or the remote operator.
4.6 Test trials shall produce enough successful repetitions to demonstrate the reliability of the system capability or the remote
operator proficiency. proficiency to the desired level of statistical significance (see Section 9). A complete trial of 10 to 30
repetitions should take 10 to 30 min to complete. When measuring system capabilities, it is important to allow enough time to
capture a complete trial with an expert operator. When measuring operator proficiency, it is important to limit the time of the trial
so that expert operators have ample time to perform a statistically significant set of repetitions while novice operators are not
excessively fatigued. There are three metrics to consider when calculating the results of a test trial. They should be considered in
the following order of importance: completeness score, reliability, and efficiency. The results from open, rectangular confinement,
and square confinement configurations are not comparable because they represent different difficulties and clearances.
5. Significance and Use
5.1 This test method is part of an overall suite of related test methods that provide repeatable measures of robotic system
mobilitymaneuvering and remote operator proficiency. The align ground contacts with parallel rails test challenges robotic system
locomotion, operator control, effective camera positioning, chassis shape variability (if available), and remote situational
awareness by the operator. As such, the align ground contacts with parallel rails test can be used to represent situations where
E3310/E3310M − 22
hazards must be avoided by the robot (for example, debris, puddles) surrounding a path in the environment, highlighting situational
awareness demands on the operator while controlling the robot.
5.2 The scale of the apparatus can vary to provide different constraints representative of typical intended deployment
environments. For example, the three configurations can be representative of repeatable complexity for unobstructed environments
(open configuration), relatively open parking lots with spaces between cars (rectangular confinement configuration), or within bus,
train, or plane aisles, or dwellings with hallways and doorways (square confinement configuration).
5.3 The test apparatuses are low cost and easy to fabricate so they can be widely replicated. The procedure is also simple to
conduct. This eases comparisons across various testing locations and dates to determine best-in-class systems and operators.
5.4 Evaluation—This test method can be used in a controlled environment to measure baseline capabilities. The parallel rails
apparatus can also be embedded into operational training scenarios to measure degradation due to uncontrolled variables in
lighting, weather, radio communications, GPS accuracy, etc.
5.5 Procurement—This test method can be used to identify inherent capability trade-offs in systems, make informed purchasing
decisions, and verify performance during acceptance testing. This aligns requirement specifications and user expectations with
existing capability limits.
5.6 Training—This test method can be used to focus operator training as a repeatable practice task or as an embedded task within
training scenarios. The resulting measures of remote operator proficiency enable tracking of perishable skills over time, along with
comparisons of performance across squads, regions, or national averages.
5.7 Innovation—This test method can be used to inspire technical innovation, demonstrate break-through capabilities, and measure
the reliability of systems performing specific tasks within an overall mission sequence. Combining or sequencing multiple test
methods can guide manufacturers toward implementing the combinations of capabilities necessary to perform essential mission
tasks.
6. Apparatus
6.1 The apparatus consists of subfloors, walls (only for rectangular confinement and square confinement configurations), pallets,
and the parallel rails (see Fig. 2). The main apparatus dimension to consider is the minimumapparatus clearance width (W) for the
robot. The minimum clearance width should be chosen to robot, which can be set to 240 cm [96 in.] with 62.5 cm [1 in.] tolerance,
120 cm [48 in.] with 62.5 cm [1 in.] tolerance, 60 cm [24 in.] with 61.3 cm [0.5 in.] tolerance, or 30 cm [12 in.] with 61.3 cm
[0.5 in.] tolerance. The dimension chosen for W should represent the intended deployment environment or should be based on the
size of the robot, or both. The minimum clearance width is typically set to 120 cm [4 ft], 60 cm [2 ft], or 30 cm [1 ft] to efficiently
use available construction materials, although other apparatus sizes can be used both (that is, the robot shall be able to maneuver
within the selected dimensions of the apparatus). All apparatus dimensions scale proportionally with W (see Fig. 3). All apparatus
dimensions scale proportionally with the minimum clearance width (see and Fig. 4). For example, the length of the parallel rails
is 2W, and the length of the apparatus is either 4W (square confinement configuration) or 6W (rectangular confinement and open
configurations). The equipment required to perform this test method includes the apparatus and a timer. Resulting data from a
specific minimum clearance width of the apparatus is not comparable to data from other apparatuses with different minimum
clearance widths.
6.2 The apparatus consists of two symmetrical approach/exit areas on either side of the parallel rails. There are three configurations
of the apparatus: open, rectangular confinement, and square confinement (see Fig. 3). The selection of apparatus configuration
should correspond to intended deployment environment. The open configuration does not use walls in the approach areas on either
side of the parallel rails, allowing for unobstructed robot movement. The approach areas in the rectangular confinement
configuration measure 2W by 1W and are bounded by walls taller than the robot to obstruct robot movement. The approach areas
in the square confinement configuration measure 1W by 1W and are bounded by walls taller than the robot to further obstruct robot
movement. Resulting data from a specific configuration of the apparatus is not comparable to data from other apparatuses with
different configurations.
6.3 Parallel Rails—The parallel rails are constructed of two pieces of dimensional lumber positioned on the ground parallel to
each other with variable distance between them, and attached to the side of two pallets on either end of the apparatus. The parallel
rails are contained within a 2W length of the apparatus (see Fig. 5). The separation distance (D) between the parallel rails
E3310/E3310M − 22
FIG. 2 View of a Test Apparatus with Labeled Components
FIG. 3 The Testing Apparatus is Scalable to Represent Different Environments
corresponds to the distance between the centerline of each parallel rail. D is set to match the overall width of the robot’s ground
contacts, meaning the outermost edges of the robot’s locomotion system (for example, wheels, tracks, treads) that make contact
with the ground during traversal (see Fig. 5). The height and width of the parallel rails (H) is relative to the scale of the apparatus
(see Table 1).
6.4 Pallet—Each pallet is constructed of dimensional lumber and OSB or similar material on the surface and is fabricated to fit
E3310/E3310M − 22
FIG. 4 Top View of a Test Apparatus Showing the Dimensions and Labeled Open, Rectangular Confinement,
and Square Confinement Approach/exit Areas
FIG. 5 View of the Apparatus Showing (A) Dimensions of the Parallel Rails and (B) Setting the Separation Distance of the Parallel Rails
to match the Overall Width of the Robot’s Ground Contacts
E3310/E3310M
...