ASTM E2853/E2853M-22

This document is not an ASTM standard and is intended only to provide the user of an ASTM standard an indication of what changes have been made to the previous version. Because
it may not be technically possible to adequately depict all changes accurately, ASTM recommends that users consult prior editions as appropriate. In all cases only the current version
of the standard as published by ASTM is to be considered the official document.
Designation: E2853 − 12 (Reapproved 2021) E2853/E2853M − 22
Standard Test Method for
Evaluating EmergencyGround Response Robot Capabilities:
Human-System Interaction (HSI): Search Tasks: Random
Mazes with Complex TerrainSearch Tasks
This standard is issued under the fixed designation E2853;E2853/E2853M; the number immediately following the designation indicates
the year of original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last
reapproval. A superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
INTRODUCTION
The robotics community needs ways to measure whether a particular robot is capable of performing
specific missions in complex, unstructured, and often hazardous environments. These missions require
various combinations of elemental robot capabilities. Each capability can be represented as a test
method, with an associated apparatus, to provide tangible challenges for various mission requirements
and performance metrics to communicate results. These test methods can then be combined and
sequenced to evaluate essential robot capabilities and remote operator proficiencies necessary to
successfully perform intended missions.
The ASTM International Standards Committee on Homeland Security Applications (E54) specifies
these standard test methods to facilitate comparisons across different testing locations and dates for
diverse robot sizes and configurations. These standards support robot researchers, manufacturers, and
user organizations in different ways. Researchers use the standards to understand mission
requirements, encourage innovation, and demonstrate break-through capabilities. Manufacturers use
the standards to evaluate design decisions, integrate emerging technologies, and harden systems.
Emergency responders and soldiers use them to guide purchasing decisions, align deployment
expectations, and focus training with standard measures of operator proficiency. Associated usage
guides describe how these standards can be applied to support various objectives.
Several suites of standards address these elemental capabilities including maneuvering, mobility,
dexterity, sensing, energy, communications, durability, proficiency, autonomy, and logistics. This
standard is part of the Human-System Interaction (HSI) suite of test methods.
1. Scope
1.1 Purpose: This test method is intended for remotely operated ground robots operating in complex, unstructured, and often
hazardous environments. It specifies the apparatuses, procedures, and performance metrics necessary to measure the capability of
a robot to maneuver and search throughout an environment to inspect objects of interest while negotiating complex terrain. This
test method is one of several related human-system interaction tests that can be used to evaluate overall system capabilities.
1.1.1 The purpose of this test method, as a part of a suite of human-system interactions (HSI) test methods, is to quantitatively
evaluate a teleoperated ground robot’s (see Terminology E2521) capability of searching in a maze.
This test method is under the jurisdiction of ASTM Committee E54 on Homeland Security Applications and is the direct responsibility of Subcommittee E54.09 on
Response Robots.
Current edition approved Jan. 1, 2021May 1, 2022. Published January 2021July 2022. Originally approved in 2012. Last previous edition approved in 20122021 as
E2853 – 12.E2853 – 12 (2021). DOI: 10.1520/E2853-12R21.10.1520/E2853_E2853M-22.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
E2853/E2853M − 22
FIG. 12 HSI: Search Tasks: Random Maze IllustrationExample Linear Inspection Rail Apparatus shown at an Angle (top) and from Over-
head (bottom)
1.1.2 Teleoperated robots shall possess a certain set of HSI capabilities to suit critical operations such as emergency responses,
including enabling the operators to search for required targets. A passage that forms on complex terrains and possesses complex
and visually similar branches is a type of environments that exists in emergency response and other robotically applicable
situations. The complexity often poses challenges for the operators to teleoperate the robots to conduct searches. This test method
is based on a standard maze and specifies metrics and a procedure for testing the search capability.
1.1.3 Emergency response robots shall enable the operator to handle many types of tasks. The required HSI capabilities include
search and navigation on different types of terrains, passages, and confined spaces. Standard test methods are required to evaluate
whether candidate robots meet these requirements.
1.1.4 ASTM E54.08.01 Task Group on Robotics specifies a HSI test suite, which consists of a set of test methods for evaluating
these HSI capability requirements. This random maze searching test method is a part of the HSI test suite. The apparatuses
associated with the test methods challenge specific robot capabilities in repeatable ways to facilitate comparison of different robot
models as well as particular configurations of similar robot models. (See Fig. 1.)
1.1.5 The test methods quantify elemental HSI capabilities necessary for ground robots intended for emergency response
applications. As such, based on their particular capability requirements, users of this test suite can select only the applicable test
methods and can individually weight particular test methods or particular metrics within a test method. The testing results should
collectively represent a ground robot’s overall HSI capability. The test results can be used to guide procurement specifications and
acceptance testing for robots intended for emergency response applications.
NOTE 1—The teleoperation performance is affected by the robot’s as well as the operator’s capabilities. Among all the standard test methods that ASTM
E54.08.01 Task Group on Robotics has specified, some depend more on the former while the others on the latter, but it would be extremely hard to totally
isolate the two factors. This HSI test suite is specified to focus on evaluating the operator’s capabilities of interacting with the robotic system, whereas
a separately specified sensor test suite, including Test Method E2566, focuses on the robots’ sensing capabilities.
NOTE 2—As robotic systems are more widely applied, emergency responders might identify additional or advanced HSI capability requirements to help
them respond to emergency situations. They might also desire to use robots with higher levels of autonomy, beyond teleoperation to help reduce their
workload—see NIST Special Publication 1011-II-1.0. Further, emergency responders in expanded emergency response domains might also desire to apply
robotic technologies to their situations, a source for new sets of requirements. As a result, additional standards within the suite would be developed. This
standard is, nevertheless, standalone and complete.
E2853/E2853M − 22
1.2 The robotic system typically includes a remote operator in control of all functionality, so an onboard camera and remote
operator display are typically required. Assistive features or autonomous behaviors may improve the effectiveness or efficiency of
the overall system.
1.3 Different user communities can set their own thresholds of acceptable performance within this test method for various mission
requirements.
1.4 Performing Location—This test method shallmay be performed in a testing laboratory or the field where the specified
apparatusanywhere the specified apparatuses and environmental conditions are can be implemented.
1.5 Units—The values stated in SI units are to be regarded as the standard. The values given in parentheses are not precise
mathematical conversions to inch-pound units. They are close approximate equivalents for the purpose of specifying material
dimensions that are readily available to avoid excessive fabrication costs of test apparatuses while maintaining repeatability and
reproducibility of the test method results. These values given in parentheses are provided for information only and are not
considered standard.International System of Units (a.k.a. SI Units) and U.S. Customary Units (a.k.a. Imperial Units) are used
throughout this test method. They are not mathematical conversions. Rather, they are approximate equivalents in each system of
units to enable use of readily available materials in different countries. The differences between the stated dimensions in each
system of units are insignificant for the purposes of comparing test method results, so each system of units is separately considered
standard within this test method.
1.6 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility
of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of
regulatory limitations prior to use.
1.7 This international standard was developed in accordance with internationally recognized principles on standardization
established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued
by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
2. Referenced Documents
2.1 ASTM Standards:
C33/C33M Specification for Concrete Aggregates
C144 Specification for Aggregate for Masonry Mortar
E2521 Terminology for Evaluating Response Robot Capabilities
E2566 Test Method for Evaluating Response Robot Sensing: Visual Acuity
E2592 Practice for Evaluating Response Robot Capabilities: Logistics: Packaging for Urban Search and Rescue Task Force
Equipment Caches
E2826/E2826M Test Method for Evaluating Response Robot Mobility Using Continuous Pitch/Roll Ramp Terrains
E2827/E2827M Test Method for Evaluating Response Robot Mobility Using Crossing Pitch/Roll Ramp Terrains
E2828/E2828M Test Method for Evaluating Response Robot Mobility Using Symmetric Stepfields Terrains
E2991/E2991M Test Method for Evaluating Response Robot Mobility: Traverse Gravel Terrain
E2992/E2992M Test Method for Evaluating Response Robot Mobility: Traverse Sand Terrain
E3349/E3349M Test Method for Evaluating Ground Robot Capabilities and Remote Operator Proficiency: Terrains: K-Rails
2.2 Additional Documents:
National Response Framework U.S. Department of Homeland Security
NIST Special Publication 1011-I-2.0 Autonomy Levels for Unmanned Systems (ALFUS) Framework Volume I: Terminology,
Version 2.0
NIST Special Publication 1011-II-1.0 Autonomy Levels for Unmanned Systems (ALFUS) Framework Volume II: Framework
Models, Version 1.0
For referenced ASTM standards, visit the ASTM website, www.astm.org, or contact ASTM Customer Service at service@astm.org. For Annual Book of ASTM Standards
volume information, refer to the standard’s Document Summary page on the ASTM website.
Available from Federal Emergency Management Agency (FEMA), P.O. Box 10055, Hyattsville, MD 20782-8055, http://www.fema.gov/emergency/nrf/. Visual acuity
targets available here: https://drive.google.com/file/d/1sUsX4rlm24LqcEe3ARNsXyBYfgyDjw0z/edit.
Available from National Institute of Standards and Technology (NIST), 100 Bureau Dr., Stop 1070, Gaithersburg, MD 20899-1070, http://www.nist.gov/el/isd/ks/
autonomy_levels.cfm. Dennis D. Leber, Leticia Pibida, and Alexander L. Enders. NIST Technical Note 2045: Confirming a Performance Threshold with a Binary
Experimental Response. National Institute of Standards and Technology, July 2019.
E2853/E2853M − 22
3. Terminology
3.1 Definitions:
3.1.1 abstain, v—the action of the manufacturer or designated operator of the testing robot choosing not to enter the test. Any
decision to take such an action shall be conveyed to the administrator before the test begins. The test form shall be clearly marked
as such, indicating that the manufacturer acknowledges the omission of the performance data while the test method was available
at the test time.
3.1.1.1 Discussion—
Abstentions may occur when the robot configuration is neither designed nor equipped to perform the tasks as specified in the test
method. Practices within the test apparatus prior to testing should allow for establishing the applicability of the test method for
the given robot.
3.1.2 administrator, n—person who conducts the test—The administrator shall ensure the readiness of the apparatus, the test form,
and any required measuring devices such as stopwatch and light meter; the administrator shall ensure that the specified or required
environmental conditions are met; the administrator shall notify the operator when the safety belay is available and ensure that the
operator has either decided not to use it or assigned a person to handle; and the administrator shall call the operator to start and
end the test and record the performance data and any notable observations during the test.
3.1.3 emergency response robot, or response robot, n—a mobile device deployable to perform operational tasks at operational
tempos to assist the operators to handle a disaster.
3.1.3.1 Discussion—
A response robot is designed to serve as an extension of the operator for gaining improved remote situational awareness and for
accomplishing the tasks remotely through the equipped capabilities. The use of a robot is designed to reduce risk to the operator
while improving effectiveness and efficiency of the mission. The desired features of a response robot include: the ability to be
rapidly deployed and remotely operated from an appropriate standoff distance and to be mobile in complex environments,
sufficiently hardened against harsh environments, reliable and field serviceable, durable and/or cost effectively disposable, and
equipped with operational safeguards.
3.1.4 fault condition, n—a certain situation or occurrence during testing whereby the robot either cannot continue without human
intervention or has performed some defined rules infraction.
3.1.4.1 Discussion—
Fault conditions include robotic system malfunction such as de-tracking, task execution problems such as excessive deviation from
a specified path, or uncontrolled behaviors and other safety violations which require administrative intervention.
3.1.5 full-ramp terrain element, n—1.2 by 1.2 m (4 by 4 ft) surface ramp with 15° slope using solid wood support posts with angle
cuts. The material used to build these elements shall be strong enough to allow the participating robots to execute the testing tasks.
3.1.5.1 Discussion—
The material that is typically used to build these elements, oriented strand board (OSB) is a commonly available construction
material. The frictional characteristics of OSB resemble that of dust covered concrete and other human improved flooring surfaces,
often encountered in emergency responses. Solid wood posts with 10 by 10 cm (4 by 4 in) cross-section dimensions typically
support the ramped surface.
3.1.5.2 Discussion—
Elements similar to this type are used, sometimes mixed and assembled in different configurations, to create various levels of
complexities for such robotic functions as orientation and traction.
3.1.6 human-scale, adj—the environments and structures typically negotiated by humans, although possibly compromised or
collapsed enough to limit human access. Also, that the response robots considered in this context are in a volumetric and weight
scale appropriate for operation within these environments.
3.1.6.1 Discussion—
No precise size and weight ranges are specified for this term. The test apparatus specifies the confined areas in which to perform
the tasks. Such constraints limit the overall sizes of robots to those considered applicable to emergency response operations.
3.1.7 maze, n—a network of passages interconnected without any repetitive order of opening and closing directions and meant to
challenge robotic navigation from the designed starting and end points.
3.1.8 operator, n—person who controls the robot to perform the tasks as specified in the test method; she/he shall ensure the
E2853/E2853M − 22
readiness of all the applicable subsystems of the robot; she/he through a designated second shall be responsible for the use of a
safety belay; and she/he shall also determine whether to abstain the test.
3.1.8.1 Discussion—
An emergency responder would be a typical operator in emergency response situations.
3.1.9 operator station, n—apparatus for hosting the operator and her/his operator control unit (OCU, see NIST Special Publication
1011-I-2.0) to teleoperate (see Terminology E2521) the robot. The operator station shall be positioned in such a manner as to
insulate the operator from the sights and sounds generated at the test apparatuses.
3.1.10 repetition, n—robot’s completion of the task as specified in the test method and readiness for repeating the same task when
required.
3.1.10.1 Discussion—
In a traversing task, the entire mobility mechanism shall be behind the START point before the traverse and shall pass the END
point to complete a repetition. A test method can specify returning to the START point to complete the task. Multiple repetitions,
performed in the same testing condition, may be used to establish the tested capability to a certain degree of statistical significance
as specified by the test sponsor.
3.1.11 test event, or event, n—a set of testing activities that are planned and organized by the test sponsor and to be held at the
designated test site(s).
3.1.12 test form, n—a collection of data fields or graphics used to record the testing results along with the associated information.
A single test form shall not be used to record the results of multiple trials.
3.1.13 test sponsor, n—an organization or individual that commissions a particular test event and receives the corresponding test
results.
3.1.14 test suite, n—designed collection of test methods that are used, collectively, to evaluate the performance of a robot’s
particular subsystem or functionality, including HSI, manipulation, sensors, energy/power, communications, logistics, safety and
operating environment, and aerial or aquatic maneuvering.
3.1.15 testing target, or target, n—a designed physical feature to be used by the testing robotic subsystem for evaluating the
subsystem capabilities. The feature may be an operationally relevant object, a notional object, or one designed specifically for
exercising the subsystem features to its full extent.
3.1.16 testing task, or task, n—a set of activities well defined in a test method for testing robots and the operators to perform in
order for the system’s capabilities to be evaluated according to the corresponding metric(s). A test method may specify multiple
tasks. A task corresponds to the associated metric(s).
3.1.17 trial, n—the number of repetitions to be performed for a test to reach required statistical significance. The repetitions may
be recorded on a single test form.
3.1 Definitions—The following terms are used in this test method and are defined in Terminology E2521: lists additional
definitions relevant to this test method.abstain, administrator or test administrator, emergency response robot or response robot,
fault condition, operator, operator station, remote control, repetition, robot, stepfield terrain element, teleoperation, test event or
event, test form, test sponsor, test suite, testing target or target, testing task or task, and trial or test trial.
3.2 Definitions of Terms Specific to This Standard:
3.2.1 alcove, n—an area measuring 1W by 1W with walls on three sides, used in the rectangular labyrinth and freeform maze test
configurations; see Fig. 1.
3.2.2 apparatus clearance width (W), n—a specification for the apparatus dimensions chosen from one of four possible
measurements, based on the intended robot deployment environment:
240 cm 6 2.5 cm tolerance [96 in. 6 1 in. tolerance], such as open and outdoor public spaces;
120 cm 6 2.5 cm tolerance [48 in. 6 1 in. tolerance], such as indoor spaces in accessibility-compliant buildings;
60 cm 6 1.3 cm tolerance [24 in. 6 0.5 in. tolerance], residences and aisles of public transportation; or
E2853/E2853M − 22
FIG. 31 Maze Floor Filled with Full-Ramp Terrain ElementsLayout and Measurements of Rooms, Hallways, and Alcoves in the Rectan-
gular Labyrinth Test Configuration
and an Example of a Layout in the Freeform Maze Test Configuration
30 cm 6 1.3 cm tolerance [12 in. 6 0.5 in. tolerance], cluttered indoor spaces, ductwork, and voids in collapsed structures.
3.2.2.1 Discussion—
The measures for these scales are nominal and do not represent the measurement of the narrowest point in the apparatus through
which the robot should pass. Consult Section 6 for the overall measurements and dimensions of the apparatus at each scale.
3.2.3 hallway, n—an area measuring 1W width by variable length which connects rooms and alcoves, used in the rectangular
labyrinth and freeform maze test configurations; see Fig. 1.
3.2.4 Landolt Ring or Landolt C, n—an optotype, or symbol, consisting of a black circular ring with a white gap or vice versa,
both with specified sizes, as defined in Test Method E2566.
3.2.5 linear inspection rail, n—a series of black and white buckets or PVC pipes with visual acuity targets, arranged at specified
angles and attached to a 0.75W long length of wood, plastic, or metal, as shown in Fig. 2; see Section 6 for more information.
3.2.6 quarter-ramp terrain element, n—inclined surface of 15° that, when projected onto the ground plane, results in a footprint
that is a square with each side equal to half of W.
3.2.7 room, n—an area measuring 2W by 2W, used in the rectangular labyrinth and freeform maze test configurations; see Fig.
1.
3.2.8 stepfield terrain element, n—discontinuous terrain type completely formed using an array of wood posts standing on end with
nominal dimensions of 10 by 10 cm [4- by 4-in.] for the cross-section and elevations of 10, 20, 30, 40, and 50 cm [4, 8, 12, 16,
and 20 in.]; the posts may be arranged to form specified topologies.
E2853/E2853M − 22
FIG. 23 Random MazeExample Visual Acuity Target and the Corresponding Ring Gap Orientations
3.2.8.1 Discussion—
This is the same definition as in Terminology E2521 – 16. This definition refers to the dimensions of the stepfield terrain elements
when W = 120 cm [96 in.].
3.2.9 visual acuity target, n—a printed graphic of nested Landolt C symbols of varying sizes and orientations; the orientation of
each C is defined by the direction of the gap in the ring out from the center.
3.2.9.1 Discussion—
This is the same type of artifact used in Test Method E2566 – 17a. See Fig. 3.
4. Summary of Test Method
4.1 The search task for this This test method is for a teleoperated robot to traverse in a specified maze to completely cover and
clear specified targets. Standard hazardous materials (HAZMAT) labels shall be used as the targets. Coverage of a target is defined
as when theperformed by a remote operator who cannot see or hear the robot within the test apparatus. The robot traverses through
a defined area over terrain of varying complexity, searching for visual acuity targets positioned at various heights and orientations
throughout the area, and inspecting and identifying as many of them as possible. The visual acuity targets are positioned in a set
of four on a linear inspection rail, with a numeric label in the center as shown in Fig. 2operator correctly detects the existence of
the target through the video images displayed on the Operator Control Unit (OCU) and conveys such existence to the administrator.
Clearance of a target is defined as when the operator correctly conveys the names . Ten or more linear inspection rails (depending
on the test configuration) are located throughout the test apparatus for a total of at least three out of the following four features
on the label:forty visual acuity targets. Three test configurations are defined (see Fig. 4color, icon, number, and words to the
administrator. When the operator correctly conveys one or two of the features, it is categorized as coverage.):
4.1.1 Rectangular Labyrinth—The robot traverses through a fabricated apparatus of a specified design. This consists of four
hallways, three rooms, and four alcoves. There are pre-defined locations that are known to the operator for one set of ten linear
inspection rails throughout the labyrinth. The robot navigates following either the left- or right-hand prescribed traversal path
through the apparatus (see Fig. 5), which is similar to performing a left or right hand wall follow.
4.1.2 Freeform Maze—The robot traverses through a fabricated maze apparatus approximately two to four times the size of the
rectangular labyrinth. This maze has multiple routes and intersections of a variable design (not specified) that consists of at least
four hallways, three rooms, and four alcoves. It also has variable locations for one or more sets of ten linear inspection rails
throughout (not pre-defined and not known to the operator), but following the prescribed heights and orientations for the linear
inspection rails as defined (see 4.4). The design of the maze layout and the locations of the linear inspection rails is to be
determined by the test sponsor, while following the selected apparatus clearance width (W) and minimum wall height (H)
measurements (see 6.3). Multiple sets of ten linear inspection rails can be used if desired. Robot navigation through the apparatus
is unrestricted, meaning there is no prescribed traversal path for the robot.
4.1.3 Embedded Scenario—The robot traverses through a real-world environment with multiple hallways and rooms (for example,
a residential or office building) or a large open space (for example, a gymnasium). The environment is approximately two to four
times the size of the rectangular labyrinth with variable locations for one or more sets of ten linear inspection rails throughout (not
pre-defined and not known to the operator), to be determined by the test sponsor. Multiple sets of ten linear inspection rails can
be used if desired. Robot navigation through the environment is unrestricted meaning there is no prescribed traversal path for the
robot.
E2853/E2853M − 22
FIG. 54 Testing Result IllustrationThe Left-Hand (top) or Right-Hand (bottom) Prescribed Traversal Path Followed by the Robot when
Performing the Rectangular Labyrinth Test Configuration
4.2 See Fig. 5 and Table 1 for a comparison of test configurations.
4.3 Based on the selected apparatus clearance width (W), minimum wall height (H) is also determined. The ratio of W:H is defined
in order to match an intended deployment environment. For example, a typical indoor environment with hallways and rooms is
defined as W = 120 cm [48 in.] and H = 240 cm [96 in.], or 1:2 ratio; a public transportation environment (for example, bus,
airplane) with narrow aisles is defined as W = 60 cm [24 in.] and H = 180 cm [72 in.], or 1:3 ratio. Using these variables, seven
different apparatus dimensional settings are defined; see Section 6 and Table 3 for more details.
4.4 The four visual acuity targets on each linear inspection rail are mounted recessed inside of buckets or pipes (see Fig. 2 and
Fig. 3) such that they are only viewable by the robot when its camera is approximately aligned/centered with the target. The
numeric label in the center of the linear inspection rail is used to identify which linear inspection rail is being inspected during
the test. The dimensions of the visual acuity targets, the buckets or pipes they are mounted in, and the rail they are attached to scale
depending on the apparatus clearance width (W). Each linear inspection rail is positioned in the apparatus according to a set of
predefined heights that are dependent on the minimum wall height (H) (ground level, 0.25H, 0.5H, 0.75H, or H) and orientations
(viewable from the front, below, or above). Additional detail is provided in Section 6.
4.5 To perform the inspection task on a linear inspection rail, the operator shall first use the robot’s camera to identify the number
label in the center of the linear inspection rail (for example, 1, 2, 3). Then they shall proceed to inspect the visual acuity targets
that are viewable by the robot’s camera, moving the robot and manipulators if necessary. Some targets may not be able to be
inspected due to limitations on the robot’s capability such as its camera resolution, reach of its inspection camera, or its manipulator
degrees of freedom. To successfully inspect a visual acuity target, the operator must first be able to see the entire black or white
ring inside of the colored ring (outside of the Landolt Cs) on the OCU display of the robot’s camera (see Fig. 6 for examples of
correct and incorrect alignment). The operator then must correctly discern the orientation of the gap in the Landolt Cs relative to
the top of the target (marked by a number/letter), for example, top, top-right, bottom, etc., doing so down to the smallest Landolt
C that they are able to. Three sizes of visual acuity targets (V) are available, identified by the diameter of the outer edge of the
black or white ring (inside of the colored ring): 8.3 cm [3.25 in.], 4 cm [1.5 in.], and 2.1 cm [0.8 in.]. The corresponding levels
of acuity for the available Landolt C symbols for each target size are shown in Table 2. The Landolt C symbols are labeled C1
(largest) through C5 (smallest); note that the corresponding acuity for some Landolt C symbols are marked as “N/A” due to
limitations in printing the 4 cm [1.5 in.] and 2.1 cm [0.8 in.] targets, which prevent some of the smallest Landolt Cs from being
printed legibly. The orientations observed by the operator shall be compared to an answer key after the test is complete in order
to determine the level of acuity achieved.
4.6 Terrain can vary in each test configuration. For the embedded scenario, the terrain that already exists in the environment can
be used (for example, carpet, concrete). For the rectangular labyrinth and freeform maze test configurations, several terrains are
specified below that can be used, many of which are referenced from other standards (see Fig. 7): flat flooring, k-rails, continuous
ramps, crossing ramps, symmetric stepfields, sand, and gravel. For the rectangular labyrinth, the terrain used must be consistent
throughout the entire apparatus. For the freeform maze, the terrain used can vary throughout the apparatus.
E2853/E2853M − 22
FIG. 45 Test Form ImplementationSearch Test Configurations, Shown Without Terrain
4.7 When using the rectangular labyrinth test configuration, the operator shall perform the test twice: once while navigating
through the apparatus following the left-hand prescribed traversal path and again using a right-hand prescribed traversal path (see
Fig. 4). Between the two navigation types, the linear inspection rails shall be inspected either in sequential order (right-hand, 1
to 10) or reverse order (left-hand, 10 to 1) as noted by the numbered label in the center of each linear inspection rail. The four
targets on any given linear inspection rail may be inspected in any order. If any or all targets of a linear inspection rail are not able
to be inspected by the robot and operator (for example, too high, not enough degrees of freedom in the robot’s manipulator to reach,
or not within the ability of the operator to control the robot to do so), then the operator may elect to skip those targets or that rail,
moving to the next one in the prescribed order. They may not return to a partially completed rail, nor may they inspect the linear
inspection rails out of the prescribed order. Doing so renders the test invalid. When using the freeform maze and embedded
E2853/E2853M − 22
TABLE 1 Testing Results for Random Maze Search for Robot A
robot #A
# of fea. C/R for covering a target (via clearing at least
height range trgt # R/C for clearing
cleared 1 feature)
A: 1 0 5 % / 60 % 2 % / 60 %
1.6 m – 2.4 m 0 % / 80 % 0 % / 80 %
(63 in. – 96 in.) (0 out of 5)
5 0
10 0
13 1
14 0
B: 22 % / 60 % 3 0 0 % / 60 % 5 % / 60 %
0.8 m – 1.6 m 5 % / 80 % 0 % / 80 % 1 % / 80 %
(31 ⁄2 in. – 63 in.) (2 out of 4 trgts) (0 out of 2)
6 2
8 0
12 2
C: 2 4 59 % / 60 %
0 m – 0.8 m 26 % / 80 %
(0 in. – 31 ⁄2 in.) (4 out of 5 trgts)
4 4
7 4
9 4
11 0
TABLE 1 Search Test Configurations Summary
Settings Rectangular Labyrinth Freeform Maze Embedded Scenario
Apparatus Fabricated apparatus of a prescribed Fabricated maze apparatus (two to four times Real world environment (two to
design with four hallways, three rooms, four larger than rectangular labyrinth) with four or four times larger than rectangular labyrinth);
alcoves more hallways, three or more rooms, four or residential, industrial, etc.
more alcoves
Number of linear One set of 10 for a total of 40 visual acuity One or more sets of 10 with 40 visual acuity targets per set
inspection rails targets
Locations of linear Prescribed Variable
inspection rails
Heights Prescribed per set of 10 linear inspection rails: Same as rectangular labyrinth and
and orientations of linear Four viewable from the front at ground level, 0.25H, 0.5H, and 0.75H high freeform maze with ±0.125H allowed variance
inspection rails Three viewable from above at ground level, 0.25H, and 0.5H high for each height in order to fit within what is
Three viewable from below at 0.5H, 0.75H, and H high available in the scenario
Terrain Homogeneous terrain throughout Existing scenario terrain (for example, carpet,
Terrain options: flat flooring, k-rails, continuous ramps, crossing ramps, stepfields, sand, concrete)
gravel
Route(s) Single route to dead end and back Multiple routes via intersections for navigation choices
Navigation Left- or right-hand prescribed traversal path Unrestricted; there is no prescribed traversal path for the robot
Metrics Completeness, acuity, time Completeness, acuity, time, return to start
Key: trgt – Target; fea. – Feature; C – confidence level; R – reliability level
FIG. 6 Correct Alignment is Defined as When the Operator is Able to See the Entire Black or White Outer Ring
Outside of the Landolt Cs (Inside of the Colored Ring), as shown in the Left and Middle Images
scenario test configurations, the operator’s navigation is unrestricted and does not follow a prescribed path, meaning linear
inspection rails can be inspected in any order.
4.8 A robot’s physical capabilities might affect the test operator’s abilities in performing the tasks. The test sponsor can elect to
weight the coverage metric higher over clearance to reduce the effects of the cameras and/or the lights when her/his primary
E2853/E2853M − 22
TABLE 2 Testing Results for Random Maze Search for Robot B
height range trgt # # of fea. C/R for covering
robot #B cleared
A: 1 4 32 % / 60 % 32 % / 60 %
1.6 m – 2.4 m 8 % / 80 % 8 % / 80 %
(63 in. – 96 in.) (3 out of 5 trgts) (3 out of 5 trgts)
5 4
10 0
13 0
14 4
B: 3 0 8 % / 60 % 8 % / 60 %
0.8 m – 1.6 m 6 4 1 % / 80 % 1 % / 80 %
(31 ⁄2 in. – 63 in.) 8 0 (1 out of 4 trgts) (1 out of 4 trgts)
12 0
C: 2 4 59 % / 60 % 59 % / 60 %
0 m – 0.8 m 26 % / 80 % 26 % / 80 %
(0 in. – 31 ⁄2 in.) (4 out of 5 trgts) (4 out of 5 trgts)
4 4
7 4
9 0
11 4
A
TABLE 2 Levels of Acuity Achievable for Each Target Size
Visual Acuity Target Size (V)
8.3 cm [3.25 in.] 4 cm [1.5 in.] 2.1 cm [0.8
Landolt C in.]
C1 10.3 mm [0.4 in.] 5.0 mm [0.2 in.] 2.6 mm [0.1 in.]
C2 4.1 mm [0.16 in.] 2.0 mm [0.08 in.] 1.0 mm [0.04 in.]
C3 1.6 mm [0.06 in.] 0.8 mm [0.03 in.] 0.4 mm [0.02 in.]
C4 0.7 mm [0.03 in.] 0.3 mm [0.01 in.] N/A
C5 0.3 mm [0.01 in.] N/A N/A
A
N ⁄A indicates Landolt Cs that cannot be printed legibly, meaning they cannot be inspected.
Key: trgt – Target; fea. – Feature; C – confidence level; R – reliability level
FIG. 7 Terrains that can be Utilized in the Rectangular Labyrinth and Freeform Maze Test Configurations
concern is the operator’s capability. Another way of handling the issue of the operator’s versus the robotic physical capabilities
is for the test sponsor to assign the respective reliability and confidence values for the two metrics according to the sponsor’s
emphases. SectionMetrics include (in order of priority): completeness (number of visual acuity targets inspected), acuity (visual
E2853/E2853M − 22
acuity level achieved per inspected target), time (time to complete the test), and return to start (if the robot returned to the start
point at 8 specifies these effects.the end of the test).
4.9 The testing robot shall return Test completeness is defined in the rectangular labyrinth test configuration as when the operator
determines that they have inspected all linear inspection rail targets possible (for example, if the robot is not able to physically
reach a high target, then it may not be possible to inspect that target) and returns to the start point in the apparatus. The test may
also end prematurely if the maximum test time (set by the test sponsor) is exceeded. For the freeform maze and embedded scenario
test configurations, the test is completed either (1) when the operator declares that they believe they have found and inspected all
linear inspection rail targets in the environment, (2) when the operator returns to the startingstart point at the end of the test. Thein
the apparatus and declares that they believe they have found and inspected all linear inspection rail targets in the environment, or
starting(3) point is specified the maximum test time (as set by the test sponsor and is not notified to the operator until at the
beginning of the test.sponsor) is exceeded. Setting a maximum test time as criteria for a successful test in the rectangular labyrinth,
freeform maze, or embedded scenario test configurations is optional.
4.4 Teleoperation shall be used from the operator station specified by the administrator to test the robots using an OCU provided
by the operator. The operator station shall be positioned and implemented in such a manner as to insulate the operator from the
sights and sounds generated at the test apparatus.
NOTE 3—Separate, autonomous search test methods will be separately specified in the future as per community requirements. This standard is,
nevertheless, standalone and complete.
4.10 The operator is allowed to practice before the test. She/he is also allowed to abstain from the test before it is started. Once
the test begins, there shall be no verbal communication between the operator and the administrator regarding the performance of
a test repetition other than describing the target as seen by the operator and instructions on when to start and notifications of faults
and any safety related conditions. The operator shall have the full responsibility to determine whether and when the robot has
completed a repetition and notify the administrator accordingly. However, it is the administrator’s authority to judge the
completeness of the repetition.Potential faults include:
4.10.1 Any contact by the robot with the apparatus that requires adjustment or repair to return the apparatus to the initial condition.
If a linear inspection rail or the visual acuity targets on the rail, or both, are moved or damaged significantly by the robot during
testing, those targets can no longer be inspected and a fault is noted on the report form. If part of the apparatus (for example, walls,
terrain) is moved or damaged significantly by the robot during testing, the operator will be instructed to pause robot operation while
the test administrator repairs the apparatus and notes the fault on the report form. The test timer will also be paused until the repairs
have been made. If necessary, the robot shall be extracted from the test apparatus in order for the repair to be made, and then
returned to the position where the fault occurred to continue testing.
4.10.2 Any visual, audible, or physical interaction that assists either the robot or the remote operator. For example, if the robot
has a failure that would require it to be manually reset (for example, if the robot’s tracks fall off, then the operator would have
to enter the test apparatus to repair them), this would constitute a fault. However, if the robot has a failure that can be repaired
while the operator remains remote (for example, if the robot’s software has to be reset and this can be performed without the
operator entering the test apparatus), this would not constitute a fault.
NOTE 4—Practice within the test apparatus is allowed to establish the applicability of the robot for the test method. It allows the operator to gain familiarity
with the standard apparatus and environmental conditions. It also enables the test administrator to establish the initial apparatus setting for the test.
4.6 The test sponsor has the authority to establish the testing policy, including the robot participation, testing schedules, test site
at which this test method is implemented, associated environmental conditions, the apparatus settings, and statistical reliability and
confidence levels of the testing results.
5. Significance and Use
5.1 A main purpose of using robots in emergency response operations is to enhance the safety and effectiveness of emergency
responders operating in hazardous or inaccessible environments. The testing results of the candidate robot shall describe, in a
statistically significant way, how reliably the robot is able to perform the specified types of tasks and thus provide emergency
responders sufficiently high levels of confidence to determine the applicability of the robot.
5.1 This test method addresses robot performance requirements expressed by emergency responders and representatives from
other interested organizations. Robot performance data captured within this test method are indicative of the robotic system’s
E2853/E2853M − 22
capabilities. Having available a roster of successfully tested robots with associated performance data to guide procurement and
deployment decisions for emergency responders is consistent with the guideline of “Governments at all levels have a responsibility
to develop detailed, robust, all-hazards response plans” as stated in National Response Framework.is part of an overall suite of
related test methods that provide repeatable measures of human-system interaction capability including robotic system mobility,
dexterity, inspection, remote operator proficiency, and situational awareness. In particular, the operator control unit (OCU) design
and interface features may impact the operator’s ability to perform movement and inspection tasks with the robot.
5.2 The test apparatuses are low cost and easy to fabricate so they can be widely replicated. The procedure is also simple to
conduct. This eases comparisons across various testing locations, dates, and times to determine best-in-class systems and operators.
5.3 Evaluation—This test method is part of a test suite and is intended to provide a capability baseline for the robotic HSI
subsystems based on the identified needs of the emergency response community. Adequate performance using this test suite will
not ensure successful operation in all emergency response situations due to possible extreme operational difficulties. Rather, this
test method is intended to provide a common comparison of technologies against a reasonable simulation of emergency response
environments and to provide quantitative performance data to emergency response organizations to aid in choosing appropriate
systems. This standard is also intended to encourage development of improved and innovative communications systems for use
on emergency response robots. can be used in a controlled environment to measure baseline capabilities. It can also be embedded
into operational training scenarios to measure degradation due to uncontrolled variables in lighting, weather, radio
communications, GPS accurac
...