ASTM E2854/E2854M-21

This document is not an ASTM standard and is intended only to provide the user of an ASTM standard an indication of what changes have been made to the previous version. Because
it may not be technically possible to adequately depict all changes accurately, ASTM recommends that users consult prior editions as appropriate. In all cases only the current version
of the standard as published by ASTM is to be considered the official document.
Designation: E2854 − 12 E2854/E2854M − 21
Standard Test Method for
Evaluating Emergency Response Robot Capabilities: Radio
Communication:Communications Line-of-Sight Range
This standard is issued under the fixed designation E2854;E2854/E2854M; the number immediately following the designation indicates
the year of original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last
reapproval. A superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
INTRODUCTION
The robotics community needs ways to measure whether a particular robot is capable of performing
specific missions in complex, unstructured, and often hazardous environments. These missions require
various combinations of elemental robot capabilities. Each capability can be represented as a test
method with an associated apparatus to provide tangible challenges for various mission requirements
and performance metrics to communicate results. These test methods can then be combined and
sequenced to evaluate essential robot capabilities and remote operator proficiencies necessary to
successfully perform intended missions.
The ASTM International Standards Committee on Homeland Security Applications (E54) specifies
these standard test methods to facilitate comparisons across different testing locations and dates for
diverse robot sizes and configurations. These standards support robot researchers, manufacturers, and
user organizations in different ways. Researchers use the standards to understand mission
requirements, encourage innovation, and demonstrate break-through capabilities. Manufacturers use
the standards to evaluate design decisions, integrate emerging technologies, and harden systems.
Emergency responders and soldiers use them to guide purchasing decisions, align deployment
expectations, and focus training with standard measures of operator proficiency. Associated usage
guides describe how these standards can be applied to support various objectives.
Several suites of standards address these elemental capabilities including maneuvering, mobility,
dexterity, sensing, energy, communications, durability, proficiency, autonomy, and logistics. This
standard is part of the communications suite of test methods.
1. Scope
1.1 Purpose: This test method is intended for remotely operated ground robots using radio communications to transmit real-time
data between a robot and its remote operator interface. This test method measures the maximum line-of-sight radio
communications distance at which a robot can maintain omnidirectional steering, speed control, precise stopping, visual acuity, and
other functionality. This test method is one of several related radio communication tests that can be used to evaluate overall system
capabilities.
1.1.1 The purpose of this test method, as a part of a suite of radio communication test methods, is to quantitatively evaluate a
teleoperated robot’s (see Terminology E2521) capability to perform maneuvering and inspection tasks in a line-of-sight
environment.
This test method is under the jurisdiction of ASTM Committee E54 on Homeland Security Applications and is the direct responsibility of Subcommittee E54.09 on
Response Robots.
Current edition approved Feb. 1, 2012Jan. 1, 2021. Published April 2012June 2021. Originally approved in 2012. Last previous edition approved in 2012 as E2854 – 12.
DOI: 10.1520/E2854-12.10.1520/E2854_E2854M-21.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
E2854/E2854M − 21
Left: The line-of-sight range test method uses an airstrip or paved road with robot test stations placed every 100 m (330 ft) along the centerline. Right: Robot test stations
are prototyped with targets on the barrels for visual inspection tasks and circular paths for maneuvering tasks.Overview of the test site showing a roadway, airstrip, or
parking lot with a centerline and measured incremental distances between the omnidirectional robot tasks and a movable remote operator interface.
FIG. 1 Test Fabrication at An Air StripOverview of the Test Site
1.1.2 Robots shall possess a certain set of radio communication capabilities, including performing maneuvering and inspection
tasks in a line-of-sight environment, to suit critical operations for emergency responses. The capability for a robot to perform these
types of tasks in unobstructed areas down range is critical for emergency response operations. This test method specifies a standard
set of apparatuses, procedures, and metrics to evaluate the robot/operator capabilities for performing these tasks.
1.1.3 Emergency response robots shall be able to operate remotely using the equipped radios in line-of-sight (LOS) environments,
in non-line-of-sight (NLOS) environments, and for signal penetration through such impediments as buildings, rubbles, and tunnels.
Additional capabilities include operating in the presence of electromagnetic interference and providing link security and data
logging. Standard test methods are required to evaluate whether candidate robots meet these requirements.
1.1.4 ASTM E54.08.01 Task Group on Robotics specifies a radio communication test suite, which consists of a set of test methods
for evaluating these communication capabilities. This line-of-sight range test method is a part of the radio communication test suite.
The apparatuses associated with the test methods challenge specific robot capabilities in repeatable ways to facilitate comparison
of different robot models as well as particular configurations of similar robot models.
1.1.5 This test method establishes procedures, apparatuses, and metrics for specifying and testing the capability of radio (wireless)
links used between the operator station and the testing robot in a line-of-sight environment. These links include the command and
control channel(s) and video, audio, and other sensor data telemetry.
1.1.6 This test method is intended to apply to ground based robotic systems and small unmanned aerial systems (sUAS) capable
of hovering to perform maneuvering and inspection tasks down range for emergency response applications.
1.1.7 This test method specifies an apparatus that is an essentially clear radio frequency channel for testing. Fig. 1 provides an
illustration.
NOTE 1—Frequency coordination and interoperability are not addressed in this standard. These issues should be resolved by the affected agencies (Fire,
Police, and Urban Search and Rescue) and written into the Standard Operating Procedures (SOPs) that guide the responses to emergency situations.
1.1.8 The radio communication test suite quantifies elemental radio communication capabilities necessary for robots intended for
emergency response applications. As such, based on their particular capability requirements, users of this test suite can select only
the applicable test methods and can individually weight particular test methods or particular metrics within a test method. The
testing results should collectively represent a ground robot’s overall radio communication capability. These test results can be used
to guide procurement specifications and acceptance testing for robots intended for emergency response applications.
NOTE 2—As robotic systems are more widely applied, emergency responders might identify additional or advanced robotic radio communication
capability requirements to help them respond to emergency situations. They might also desire to use robots with higher levels of autonomy, beyond
teleoperation to help reduce their workload—see NIST Special Publication 1011-II-1.0. Further, emergency responders in expanded emergency response
E2854/E2854M − 21
domains might also desire to apply robotic technologies to their situations, a source for new sets of requirements. As a result, additional standards within
the suite would be developed. This standard is, nevertheless, standalone and complete.
1.2 A remote operator is in control of all functionality, so an onboard camera and remote operator display are typically required.
Assistive features or autonomous behaviors may improve the effectiveness or efficiency of the overall system.
1.3 Different user communities can set their own thresholds of acceptable performance within this test method to address various
mission requirements.
1.4 Performing Location—This test method shallmay be performed in a testing laboratory or the field where the specified
apparatusanywhere the specified apparatuses and environmental conditions are can be implemented.
1.5 Units—The values stated in SI units shall be the standard. The values given in parentheses are not precise mathematical
conversions to inch-pound units. They are close approximate equivalents for the purpose of specifying material dimensions or
quantities that are readily available to avoid excessive fabrication costs of test apparatuses while maintaining repeatability and
reproducibility of the test method results. These values given in parentheses facilitate testing but are not considered
standard.International System of Units (a.k.a. SI Units) and U.S. Customary Units (a.k.a. Imperial Units) are used throughout this
document. They are not mathematical conversions. Rather, they are approximate equivalents in each system of units to enable the
use of readily available materials in different countries. The differences between the stated dimensions in each system of units are
insignificant for the purposes of comparing test method results, so each system of units is separately considered standard within
this test method.
1.6 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility
of the user of this standard to establish appropriate safety safety, health, and healthenvironmental practices and determine the
applicability of regulatory limitations prior to use.
1.7 This international standard was developed in accordance with internationally recognized principles on standardization
established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued
by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
2. Referenced Documents
2.1 ASTM Standards:
E2521 Terminology for Evaluating Response Robot Capabilities
E2566 Test Method for Evaluating Response Robot Sensing: Visual Acuity
E2592 Practice for Evaluating Response Robot Capabilities: Logistics: Packaging for Urban Search and Rescue Task Force
Equipment Caches
E2855 Test Method for Evaluating Emergency Response Robot Capabilities: Radio Communication: Non-Line-of-Sight Range
2.2 AdditionalOther Documents:
National Response Framework, U.S. Department of Homeland Security
NIST Special Publication 1011-I-2.01011-II-1.0 Autonomy Levels for Unmanned Systems (ALFUS) Framework Volume I:
Terminology, Version 2.0I:3
NIST Special Publication 1011-II-1.01011-I-2.0 Autonomy Levels for Unmanned Systems (ALFUS) Framework Volume II:
Framework Models, Version 1.0I: Terminology, Version 2.04
3. Terminology
3.1 Definitions:
3.1.1 abstain, v—the action of the manufacturer or designated operator of the testing robot choosing not to enter the test. Any
decision to take such an action shall be conveyed to the administrator before the test begins. The test form shall be clearly marked
as such, indicating that the manufacturer acknowledges the omission of the performance data while the test method was available
at the test time.
For referenced ASTM standards, visit the ASTM website, www.astm.org, or contact ASTM Customer Service at service@astm.org. For Annual Book of ASTM Standards
volume information, refer to the standard’s Document Summary page on the ASTM website.
Available from Federal Emergency Management Agency (FEMA), P.O. Box 10055, Hyattsville, MD 20782-8055, http://www.fema.gov/emergency/nrf/.
Available from National Institute of Standards and Technology (NIST), 100 Bureau Dr., Stop 1070, Gaithersburg, MD 20899-1070, http://www.nist.gov/el/isd/ks/
autonomy_levels.cfm.
E2854/E2854M − 21
3.1.1.1 Discussion—
Abstentions may occur when the robot configuration is neither designed nor equipped to perform the tasks as specified in the test
method. Practices within the test apparatus prior to testing should allow for establishing the applicability of the test method for
the given robot.
3.1.2 administrator, n—person who conducts the test—The administrator shall ensure the readiness of the apparatus, the test form,
and any required measuring devices such as stopwatch and light meter; the administrator shall ensure that the specified or required
environmental conditions are met; the administrator shall notify the operator when the safety belay is available and ensure that the
operator has either decided not to use it or assigned a person to handle; and the administrator shall call the operator to start the
test and record the performance data and any notable observations during the test.
3.1.3 emergency response robot, or response robot, n—a remotely deployed device intended to perform operational tasks at
operational tempos to assist the operators to handle a disaster.
3.1.3.1 Discussion—
A response robot is designed to serve as an extension of the operator for gaining improved remote situational awareness and for
accomplishing the tasks remotely through the equipped capabilities. The use of a robot is designed to reduce risk to the operator
while improving effectiveness and efficiency of the mission. The desired features of a response robot include: rapid deployment;
remote operation from an appropriate standoff distance; mobile in complex environments; sufficiently hardened against harsh
environments; reliable and field serviceable; durable and/or cost effectively disposable; and equipped with operational safeguards.
3.1.4 fault condition, n—a certain situation or occurrence during testing whereby the robot either cannot continue without human
intervention or has performed some defined rules infraction.
3.1.4.1 Discussion—
Fault conditions include robotic system malfunctions such as de-tracking, task execution problems such as excessive deviation
from a specified path, or uncontrolled behaviors and other safety violations which require administrative intervention.
3.1.5 human-scale, adj—used to indicate that the objects, terrains, or tasks specified in this test method are in a scale consistent
with the environments and structures typically negotiated by humans, although possibly compromised or collapsed enough to limit
human access. Also, that the response robots considered in this context are in a volumetric and weight scale appropriate for
operation within these environments.
3.1.5.1 Discussion—
No precise size and weight ranges are specified for this term. The test apparatus constrains the environment in which the tasks are
performed. Such constraints, in turn, limit the types of robots to be considered applicable to emergency response operations.
3.1.6 line-of-sight communications, n—propagating electromagnetic energy with a direct path between a transmitting radio
antenna and a receiving radio antenna which are in visual contact with each other with no obstructions between them. In the ideal
case, the only paths that the radio waves can take in the line-of-sight case are the direct path either between the transmitter and
receiver or a path that corresponds to a single reflection of the radio wave off of the ground before it encounters the receiving
antenna.
3.1.7 non-line-of-sight communications, n—propagating electromagnetic energy with no direct path between a transmitting radio
antenna and a receiving radio antenna which are not in visual contact with each other due to obstructions between them. Radio
waves propagate between the transmitting and the receiving antennas via reflections off structures, diffraction around structures,
and/or passage through structures with attenuation.
3.1.8 operator, n—person who controls the robot to perform the tasks as specified in the test method; she/he shall ensure the
readiness of all the applicable subsystems of the robot; she/he through a designated second shall be responsible for the use of a
safety belay; and she/he shall also determine whether to abstain the test.
3.1.8.1 Discussion—
An emergency responder would be a typical operator in emergency response situations.
3.1.9 operator station, n—apparatus for hosting the operator and her/his operator control unit (OCU, see NIST Special Publication
1011-I-2.0) to teleoperate (see Terminology E2521) the robot. The operator station shall be positioned in such a manner as to
insulate the operator from the sights and sounds generated at the test apparatuses.
E2854/E2854M − 21
3.1.10 radio interference, n—adverse effect on the transfer of data when unrelated external signals are received by a robot receiver
or an operator station receiver.
3.1.10.1 Discussion—
In licensed frequency bands such as those used by the public safety community, each radio transmitter and receiver is assigned
a unique frequency channel typically with limits on power emissions. Some radio systems are designed to work effectively when
multiple systems operate in the same frequency band at the same time. Many of these systems can be found in the unlicensed
Industrial, Scientific, and Medical (ISM) frequency bands.
3.1.11 repetition, n—robot’s completion of the task as specified in the test method and readiness for repeating the same task when
required.
3.1.11.1 Discussion—
In a traversing task, the entire mobility mechanism shall be behind the START point before the traverse and shall pass the END
point to complete a repetition. A test method can specify returning to the START point to complete the task. Multiple repetitions,
performed in the same test condition, may be used to establish the tested capability to a certain degree of statistical significance
as specified by the test sponsor.
3.1.12 test event, or event, n—a set of testing activities that are planned and organized by the test sponsor to be held at the one
or multiple designated test site(s).
3.1.13 test form, n—a collection of data fields or graphics used to record the testing results along with the associated information.
A single test form shall not be used to record the results of multiple trials.
3.1.14 test sponsor, n—an organization or individual that commissions a particular test event and receives the corresponding test
results.
3.1.15 test suite, n—a designed collection of test methods that are used collectively to evaluate the performance of a robot’s
particular subsystem or functionality, including mobility, manipulation, sensors, energy/power, communications, human-system
interaction (HSI), logistics, safety and operating environment, and aerial or aquatic maneuvering.
3.1.16 testing target, or target, n—a designed physical feature to be used by the testing robotic subsystem for evaluating the
subsystem capabilities. The feature may be an operationally relevant object, a notional object, or one designed specifically for
exercising the subsystem features to its full extent.
3.1.17 testing task, or task, n—a set of activities well defined in a test method for testing robots and the operators to perform in
order for the system’s capabilities to be evaluated according to the corresponding metric(s). A test method may specify multiple
tasks. A task corresponds to the associated metric(s).
3.1.18 trial, n—the number of repetitions to be performed for a test to reach required statistical significance. The repetitions may
be recorded on a single test form.
3.1 The following terms are used in this test method and are defined in Terminology E2521: lists additional definitionsemergency
response robot relevantor to thisresponse robot, testfault condition, method.Landolt C,line-of-sight communications,non-line-of-
sight communications,optotype, and radio interference.
3.2 The following terms are used in this test method and are defined in ALFUS Framework Volume I:3: autonomous,autonomy,
level of autonomy,operator control unit (OCU), and semi-autonomous, and remote teleoperation.
4. Summary of Test Method
4.1 This test method uses remote maneuvering and inspection tasks to measure the is intended for remotely operated ground robots
using radio communications to transmit real-time data between a robot and its remote operator interface. This test method specifies
robot maneuvering and camera pointing tasks performed from designated standoff distances between the robot and remote operator
interface (see Fig. 1line-of-sight range of a robot using its equipped radio communication system. This test represents the least
complicated propagation environment with ground effects that will be encountered by radio linked robotic systems.). This test
E2854/E2854M − 21
Left) The robot maneuvering tasks include a circular robot path with 3 m [10 ft] radius for the robot to straddle and follow.
Middle) The center buckets are perpendicular and limit the viewing angles of the interior targets.
Right) Each bucket target has an inscribed ring (shown as green) to evaluate successful alignment along with five increasingly small concentric ring gap orientations
to evaluate visual acuity.
FIG. 2 Robot Maneuvering Tasks
method measures the maximum line-of-sight radio communications range at which a robot can complete omnidirectional tasks
including continuous steering, speed control, precise stopping, visual acuity, and other functionality. This test method is conducted
in an environment with no radio frequency interference and minimal radio propagation effects. The same test can be conducted
at any operationally significant environment (with representative radio interference) as a practical measure of line-of-sight radio
communications range.
4.2 The test course shall be a flat paved This test method is conducted on a straight and flat surface at least 1000 m (3300 ft) long
by 20 m (65 ft) wide with a centerline robot path. A minimum of 50 m (165 ft) on each side 6 m [20 ft] wide and longer than the
maximum radio communications range of the robotic system being evaluated, or longer than the operationally significant range
of the intended application. There must be no obstructions on the paved surface or radio reflective metal objects within 50 m [165
ft] of the centerline robot path shall be clear of any obstructions or reflecting objects to minimize multi-path effects.to minimize
effects from multi-path radio transmissions. A roadway, airstrip, or parking lot can be used depending on the overall length required
(see Fig. 2).
4.3 Test stations shall be placed every 100 m (330 ft) down range along the centerline. Each test station consists of eight visual
and audio targets for inspection tasks along with circular robot paths marked on the ground for maneuvering tasks.
4.4 At each test station, the robot shall perform a maneuvering task to follow the circular path to locate each of the visual and audio
targets.
4.3 The visual and audio targets shall be identified using the robot’s forward facing cameras, requiringmaneuvering tasks require
the robot to straddle and follow a circular path marked on the ground with 3 m [10 ft] radius to demonstrate continuous steering
and speed control. The robot also aligns with four perpendicular buckets in the center using a designated forward-facing camera
on or over the robot chassis. These tasks require the robot to face all four compassdifferent directions relative to the direction of
travel operator interface to ensure that there are no directionality issues with transmitting or receiving communication signals. Each
recessed bucket target has an inscribed ring with a limited viewing angle to evaluate successful alignment. A 5-point score records
successful completion of the robot maneuvering tasks (see Fig. 3 and Fig. 4).
4.4 The robot’s line-of-sight range capability is measured as the maximum distance downrange at which the robot completes all
the tasks at a test station to verify the functionality of line-of-sight control, video and audio transmissions. visual acuity tasks
require identifying up to five increasingly small concentric ring gap orientations in each bucket. A separate 5-point acuity score
per target across four different targets totals 20 points for overall acuity.
4.5 Teleoperation shall be used from the operator station specified by the administrator to test theThere are four performance
metrics to consider when calculating the results of a test trial. They should be considered in the following order of importance:
robots using an OCUline-of-sight radio communications range,reliability, provided byaverage visual acuity, the operator. The
operator station shall be positioned and implemented effıciency.in such a manner as to insulate the operator from the sights and
sounds generated at the test apparatus.
NOTE 3—Separate, autonomous radio communications test methods will be separately specified in the future as per community requirements. This
standard is, nevertheless, standalone and complete.
E2854/E2854M − 21
Left) The robot is shown successfully straddling the circular line by having at least one ground contact on both sides.
Middle) When ground contacts are touching the circular line the robot is still considered successfully straddling.
EachRight) robot test station includes a maneuvering task to follow marked circular paths around two sets off our targets facing compass directions relative to the
direction of travel.All the ground contacts are on one side of the circular line, so the task is considered unsuccessful. Alternatively, the robot could have left the circular
line toward the inside which would also be considered unsuccessful.
FIG. 23 Test StationSuccessful Straddling and Unsuccessful Attempt
A) An example of a visual acuity eye chart and an audio speaker playing a series of computer generated, single digit numbers. B) A prototype test station apparatus
shows two barrels with only four targets total and half circular robot paths. A complete test station shall have full circular paths around each barrel with four targets facing
all compass directions totaling eight total targets per test station.Successful alignment with each bucket requires visibility of a continuously inscribed ring on the interior
bottom of the bucket. The inscribed ring is shown as green with a black inner edge to increase contrast. When the inner black edge is clearly broken the robot is considered
NOT ALIGNED.
FIG. 34 Test Station ImplementationSuccessful Alignment and Unsuccessful Alignment
4.6 This test method is performed with appropriate safety precautions to mitigate any potentially dangerous robot behaviors due
to lost communications. The operator performs the maneuvering and visual acuity tasks from a standoff distance near where loss
of either control or video is evident. The test is then repeated closer to the robot along the centerline at incremental distances of
6 m [20 ft] until all omnidirectional maneuvering and visual acuity tasks are performed successfully. The maximum distance from
the remote operator interface and its co-located antenna to the center of the circle is considered the maximum line-of-sight radio
communications range.
4.7 The operator is allowed to practice before the test. She/he is also allowed to abstain from the test before it is started. Once
the test begins, there shall be no verbal communication between the operator and the administrator regarding the performance of
a test repetition other than describing the targets as seen by the operator, instructions on when to start, and notifications of faults
and any safety related conditions. The operator shall have the full responsibility to determine whether and when the robot has
completed a repetition and notify the administrator accordingly. However, it is the administrator’s authority to judge the
completeness of the repetition.Potential Faults Include:
4.7.1 Any contact by the robot with the apparatus that requires adjustment or repair to return the apparatus to the initial condition.
E2854/E2854M − 21
4.7.2 Any visual, audible, or physical interaction that assists either the robot or the remote operator.
4.7.3 Leaving the apparatus during the trial.
NOTE 4—Practice within the test area is could help establish the applicability of the robot for the given test method. It allows the operator to gain
familiarity with the test method and environmental conditions. It also helps the test administrator to establish the initial apparatus setting for the test.
4.8 The test sponsor has the authority to establish the testing policy, including the robot participation, testing schedules, test site
at which this test method is implemented, associated environmental conditions, the apparatus settings, and statistical reliability and
confidence levels of the testing results.Test trials shall produce enough successful repetitions to demonstrate the reliability of the
system capability or the remote operator proficiency. A complete trial of 10 to 30 repetitions should take 30 to 60 min to complete.
When measuring system capabilities, it is important to allow enough time to capture a complete trial with an expert operator. When
measuring operator proficiency, it is important to limit the time of the trial so that novice and expert operators are similarly
fatigued.
4.9 Various other operationally significant targets can be incorporated into this test method to evaluate color acuity, thermal acuity,
audio acuity, latency, signal/packet loss, etc.
5. Significance and Use
5.1 A main purpose of using robots in emergency response operations is to enhance the safety and effectiveness of emergency
responders operating in hazardous or inaccessible environments. The testing results of the candidate robot shall describe, in a
statistically significant way, how reliably the robot is able to perform the specified types of tasks and thus provide emergency
responders sufficiently high levels of confidence to determine the applicability of the robot.
5.1 This test method addresses robot performance requirements expressed by emergency responders and representatives from
other interested organizations. The performance data captured within this test method are indicative of the testing robot’s
capabilities. Having available a roster of successfully tested robots with associated capabilities data to guide procurement and
deployment decisions for emergency responders is consistent with the guideline of “Governments at all levels have a responsibility
to develop detailed, robust, all-hazards response plans” as stated in National Response Framework.is part of an overall suite of
related tests that provide reproducible measures of radio communications for remotely operated robots. It measures the maximum
line-of-sight radio communications range between a robot and its remote operator interface using omnidirectional robot
maneuvering and visual acuity tasks to evaluate the degradation of essential mission capabilities due to communications latency
and loss.
5.2 This test method is inexpensive, easy to fabricate, and simple to conduct so it can be replicated widely. This enables
comparisons across various testing locations and dates to determine best-in-class system capabilities and remote operator
proficiency.
5.3 Evaluations—This test method is part of a test suite and is intended to provide a capability baseline for the robotic
communications systems based on the identified needs of the emergency response community. Adequate testing performance will
not ensure successful operation in all emergency response environments due to possible extreme communications difficulties.
Rather, this standard is intended to provide a common comparison that can aid in choosing appropriate systems. This standard is
also intended to encourage development of improved and innovative communications systems for use on emergency response
robots.can be conducted in a controlled environment with no radio frequency interference and minimal radio propagation effects
to measure baseline capabilities that can be compared widely across robotic systems. It also can be embedded into any operational
training scenario as a practical measure of line-of-sight radio communications range with additional degradation due to
uncontrolled variables such as radio frequency interference, weather, etc. The results of these scenario tests can be compared across
robotic systems only when conducted in the same environment in similar conditions. However, the results cannot be compared
reliably to results from other venues or environmental conditions due to the uncontrolled variables.
5.4 Procurement—This test method can be used to identify inherent capability trade-offs in systems, make informed purchasing
decisions, and verify performance during acceptance testing. This aligns requirement specifications and user expectations with
existing capability limits.
5.5 Training—This test method can be used to focus operator training as a repeatable practice task or as an embedded task within
E2854/E2854M − 21
training scenarios. Operators can learn system behaviors during radio communication degradation and refine techniques to mitigate
issues while performing tasks. The resulting measures of remote operator proficiency enable tracking of perishable skills over time,
along with comparisons of performance across organizations, regions, or national averages.
5.6 Innovation—The standard apparatus is specified to be easily fabricated to facilitate self-evaluation by robot developers and
provide practice tasks for emergency responders to exercise robot actuators, sensors, and operator interfaces. The standard
apparatus can also be used to support operator training and to establish operator proficiency. This test method can be used to inspire
technical innovation, demonstrate break-through capabilities, and measure the reliability of systems performing specific tasks
within an overall mission sequence. Combining or sequencing multiple tests can guide manufacturers toward implementing the
combinations of capabilities necessary to perform essential mission tasks.
5.5 Although the test method was developed first for emergency response robots, it may be applicable to other operational
domains, such as law enforcement and armed services.
6. Apparatus
6.1 Test Environment:
6.1.1 This test method is conducted in an environment with no radio frequency interference near the frequency bands used by the
robot being evaluated. This requires radio frequency monitoring equipment to ensure there is no interference from other sources.
Variants of this test method should also be performed in environments with other known or unknown radio frequency emissions
in the vicinity. Robots should be evaluated using this test method in operational scenarios with powerful radio transmitters nearby
such as emergency response vehicles, cell phone towers, and even hand-held radios as a source of potentially significant radio
interference. These test variants are less repeatable but can still provide performance comparisons for various systems tested in the
same environment at roughly the same time.
6.2 The test apparatus is a straight, flat section of airstrip, roadway or other paved asphalt or concrete surface at least 1000 m (3300
ft) long and 20 m (65 ft) wide. It shall have no obstructions or reflective objects within at least 50 m (165 ft) on either side of the
centerline. Test Site (see Fig. 1):
6.2.1 The test site must be a flat paved surface at least 6 m [20 ft] wide with an overall centerline length longer than the maximum
radio communications range of the robotic system being evaluated, or longer than the operationally significant range of the
intended application. It must be flat and straight enough to maintain an unobstructed view between the robot and the remote
operator interface antenna throughout the test. A slight incline or decline can be tolerated as long as there is no depression along
the centerline that obscures the direct line-of-sight path from the robot to the operator interface antenna. The top of the antenna
is limited to a maximum height of 2 m [6 ft] from the ground. There must also be no obstructions or reflecting metal objects within
50 m [165 ft] of the centerline to minimize effects from multi-path radio transmissions to only reflections off the ground. A
roadway, airstrip, or parking lot can be used depending on the system capabilities being evaluated as long as there are no metal
buildings, vehicles, guardrails, signs, etc. Operationally significant variants of this test method performed on absorptive ground
surfaces such as grass and with reflective metal objects in the vicinity, or with antennas mounted on vehicles or structures such
that they exceed 2 m [6 ft] in height, or combinations thereof, are less repeatable but can still provide points of comparison for
various systems tested in the same environment in similar conditions.
6.3 Test stations, specified below, shall be placed every 100 m (330 ft) down range from the operator station along the centerline
robot test path (see Fig. 1). Incremental Standoff Distances (see Fig. 1):
6.3.1 The incremental standoff distances along the centerline locate the remote operator interface relative to the center of the
circular robot tasks. Each incremental distance is 6 m [20 ft]. The overall length must extend beyond the maximum line-of-sight
radio communications range of the robotic system being evaluated. The centerline can be a roadway lane marker line, pavement
seam, or measuring tape pulled taught and secured to the ground. Each measured increment should be marked with spray chalk
or other means to clearly identify the location and distance from the center of the circular robot path.
6.4 Each test station shall have two circular robot paths marked on the ground each with a 2 m (6.5 ft) radius (see Fig. 2 and Fig.
3). The circular robot paths shall be tangent to each other, with the connection point marking the measured distance downrange
from the operator station. Markings on the circular paths shall show the location at which the robot must turn and face the targets
to identify them.Circular Robot Path (see Fig. 2):
E2854/E2854M − 21
Left) The elevation of the buckets needs to align with various size robots, so some vertical adjustment is necessary.
Middle) Stackable crates provide excellent perpendicular mounting surfaces and easy adjustment of elevation.
Right) Wood blocks inside the crates enable the buckets to be affixed to the outside surfaces with screws. Bolts, washers, and wingnuts through the bucket and crate
can also be used to enable quick set up and stowing.
FIG. 45 Example of a Test Form (Blank)Perpendicular Buckets
6.4.1 The circular robot path marked on the ground provides a 3 m [10 ft] radius line for the robot to straddle and follow. The
circle can be marked using a fixed length rope tied between a spray chalk roller wand and center stake or weight. Pull the rope
taught to ensure the radius is correct, then pull continuously outward away from the center while marking. Four additional
markings at 90-degree increments around the circle designate the locations where the robot rotates inward to align with the
perpendicular buckets. The locations for bucket A and bucket C are the intersection of the circle and the centerline with Bucket
A closer to the remote operator interface and C further from the remote operator interface. The locations for bucket B and bucket
D are 90-degrees from the centerline with bucket B to the left when viewed from the operator interface and bucket D to the right.
Use the correct alignment position of bucket B and bucket D to mark the locations.
6.5 Each test station shall have eight unique visual targets to be identified through the equipped communications channel. The
visual targets shall be placed at the center of the circular robot paths facing all four compass directions (north, south, east, and west)
relative to the direction of travel on the centerline path.Perpendicular Buckets (see Fig. 5):
6.5.1 Four perpendicular buckets are located at the center of the circular robot path to restrict the viewing angles of targets affixed
to their inside bottom surfaces. They are aligned with positions every 90-degrees around the circular robot path starting where the
circle intersects the centerline. The buckets are incrementally elevated using the crates shown or other non-metallic apparatus to
ensure different sized robots can see the entire inscribed ring when in the correct location around the circle. The buckets shown
are 7.5 L [2 gal] with 20 cm [8 in.] inner diameter so they fit letter/A4-size target stickers. White buckets are used to reflect light
and illuminate the recessed targets. Other bucket sizes and colors can work as long as the recessed targets are clearly visible to
the robot being evaluated. Each bucket is uniquely named with letters (A, B, C, D) to identify the different robot orientations for
scoring purposes. The bucket labeled “A” is viewable from the intersection of the circular robot path and the centerline closest to
the remote operator station. The other letters continue leftward or clockwise from there.
6.6 Each test station shall also have two audio sources to be identified through the equipped communications channel. The center
of each circular robot path shall have an audio source and speaker playing a continuous series of single digit numbers for the
identification task. The numbers shall be articulated using a computer-generated voice with a volume of at least 60 to 80 dB.Bucket
Targets (see Fig. 4, Fig. 5, and Fig. 6):
6.6.1 Each bucket contains a recessed target affixed to the interior bottom surface. Targets include an inscribed ring to verify robot
alignment. The inscribed ring shown is 2.5 cm [1 in.] thick and green, but any contrasting color can be used. The visual acuity
target contains five increasingly small concentric rings with various gap orientations to correctly identify. These are concentric
Landolt-C optotypes (see Test Method E2566 – 17a or later). Each ring gap can be in eight different orientations making the overall
visual acuity target randomizable with a unique answer key. The naming convention for the different gap orientations are: top (T),
top right (TR), right (R), bottom right (BR), bottom (B), bottom left (BL), left (L), top left (TL). The ring gaps are the following
sizes from largest (outer) to smallest (inner):
E2854/E2854M − 21
Left) Visual acuity targets affixed to the interior bottom of the buckets have increasingly small concentric ring gap orientations.
Right) The associated naming convention for each ring gap orientation.
FIG. 56 Example of a Test Form (Filled out)Bucket Targets
Largest Ring 1 5 % Human Vision
Gap Size = 20 mm [0.8 in.] from 3 m [10 ft]
Ring 2 11 % Human Vision
Gap Size = 8.0 mm [0.3 in.] from 3 m [10 ft]
Ring 3 27 % Human Vision
Gap Size = 3.2 mm [0.125 in.] from 3 m [10 ft]
Ring 4 67 % Human Vision
Gap Size = 1.3 mm [0.05 in.] from 3 m [10 ft]
Smallest Ring 5 175 % Human Vision
Gap Size = 0.5 mm [0.02 in.] from 3 m [10 ft]
6.7 Timer:
6.7.1 A timer is used to measure the elapsed time for the robot to perform each set of maneuvering and visual acuity tasks.
6.8 Optional Time-synced Clocks:
6.8.1 Two time-synced digital clocks can be used to detect and measure latency and loss issues due to degraded radio
communications. These latency and loss issues can last several seconds and increase the difficulty of the tasks. One time-synced
clock should be placed with the remote operator interface display and the other time-synced clock placed so it is viewable by the
robot’s camera when the robot is aligned with the buckets. The time-synced clock with the buckets needs to be large enough to
be viewable through the remote operator interface and easily comparable to the time-syn
...