ASTM E2826/E2826M-20

This document is not an ASTM standard and is intended only to provide the user of an ASTM standard an indication of what changes have been made to the previous version. Because
it may not be technically possible to adequately depict all changes accurately, ASTM recommends that users consult prior editions as appropriate. In all cases only the current version
of the standard as published by ASTM is to be considered the official document.
Designation: E2826 − 11 E2826/E2826M − 20
Standard Test Method for
Evaluating Emergency Evaluating Response Robot
Capabilities: Mobility: Confined Area Terrains: Mobility
Using Continuous Pitch/Roll RampsRamp Terrains
This standard is issued under the fixed designation E2826;E2826/E2826M; the number immediately following the designation indicates
the year of original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last
reapproval. A superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
INTRODUCTION
The robotics community needs ways to measure whether a particular robot is capable of performing
specific missions in complex, unstructured, and often hazardous environments. These missions require
various combinations of elemental robot capabilities. Each capability can be represented as a test
method with an associated apparatus to provide tangible challenges for various mission requirements
and performance metrics to communicate results. These test methods can then be combined and
sequenced to evaluate essential robot capabilities and remote operator proficiencies necessary to
successfully perform intended missions.
The ASTM International Standards Committee on Homeland Security Applications (E54) specifies
these standard test methods to facilitate comparisons across different testing locations and dates for
diverse robot sizes and configurations. These standards support robot researchers, manufacturers, and
user organizations in different ways. Researchers use the standards to understand mission
requirements, encourage innovation, and demonstrate break-through capabilities. Manufacturers use
the standards to evaluate design decisions, integrate emerging technologies, and harden systems.
Emergency responders and soldiers use them to guide purchasing decisions, align deployment
expectations, and focus training with standard measures of operator proficiency. Associated usage
guides describe how these standards can be applied to support various objectives.
Several suites of standards address these elemental capabilities including maneuvering, mobility,
dexterity, sensing, energy, communications, durability, proficiency, autonomy, and logistics. This
standard is part of the Mobility Suite of test methods.
1. Scope
1.1 Purpose: This test method is intended for remotely operated ground robots operating in complex, unstructured, and often
hazardous environments. It specifies the apparatuses, procedures, and performance metrics necessary to measure the capability of
a robot to traverse complex terrains in the form of continuous pitch/roll ramps. This test method is one of several related mobility
tests that can be used to evaluate overall system capabilities.
1.1.1 The purpose of this test method, as a part of a suite of mobility test methods, is to quantitatively evaluate a teleoperated
ground robot’s (see Terminology E2521) capability of traversing complex terrain composed of continuous pitch/roll ramps in
confined areas.
1.1.2 Robots shall possess a certain set of mobility capabilities, including negotiating complex terrains, to suit critical
operations such as emergency responses. A part of the complexity is that the environments often pose constraints to robotic
mobility to various degrees. This test method specifies apparatuses to standardize a confined areas terrain that is composed of
continuous pitch/roll ramps and that notionally represents types of terrains containing undulating slopes, existent in emergency
response and other environments. This test method also specifies procedures and metrics to standardize testing using the apparatus.
1.1.3 The test apparatuses are scalable to provide a range of lateral dimensions to constrain the robotic mobility during task
performance. Fig. 1 shows three apparatus sizes to test robots intended for different emergency response scenarios.
This test method is under the jurisdiction of ASTM Committee E54 on Homeland Security Applications and is the direct responsibility of Subcommittee E54.09 on
Response Robots.
Current edition approved July 1, 2011March 1, 2020. Published December 2011April 2020. Originally approved in 2011. Last previous edition approved in 2011 as
E2826/E2826M – 11. DOI: 10.1520/E2826-11.10.1520/E2826_E2826M-20.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
E2826/E2826M − 20
FIG. 1 Mobility: Confined Area Terrains: Overview of the Continuous Pitch/Roll Ramps ApparatusesRamp Terrain Apparatus
1.1.4 Emergency response ground robots shall be able to handle many types of obstacles and terrains. The required mobility
capabilities include traversing gaps, hurdles, stairs, slopes, various types of floor surfaces or terrains, and confined passageways.
Yet additional mobility requirements include sustained speeds and towing capabilities. Standard test methods are required to
evaluate whether candidate robots meet these requirements.
1.1.5 ASTM Task Group E54.08.01 on robotics specifies a mobility test suite, which consists of a set of test methods for
evaluating these mobility capability requirements. This continuous pitch/roll ramps terrain traversing test method is a part of the
mobility test suite. Fig. 2 shows examples of other confined area terrains, along with the traversing paths. The apparatuses
associated with the test methods challenge specific robot capabilities in repeatable ways to facilitate comparison of different robot
models as well as particular configurations of similar robot models.
1.1.6 The mobility test methods quantify elemental mobility capabilities necessary for ground robot intended for emergency
response applications. As such, users of this standard can use either the entire suite or a subset based on their particular
performance requirements. Users are also allowed to weight particular test methods or particular metrics within a test method
differently based on their specific performance requirements. The testing results should collectively represent an emergency
response ground robot’s overall mobility performance as required. These performance data can be used to guide procurement
specifications and acceptance testing for robots intended for emergency response applications.
NOTE 1—Additional test methods within the suite are anticipated to be developed to address additional or advanced robotic mobility capability
requirements, including newly identified requirements and even for new application domains.
1.2 The robotic system includes a remote operator in control of all functionality, so an onboard camera and remote operator
display are typically required. Assistive features or autonomous behaviors that improve the effectiveness or efficiency of the overall
system are encouraged.
1.3 Different user communities can set their own thresholds of acceptable performance within this test method for various
mission requirements.
1.4 Performing Location—This test method shallmay be performed in a testing laboratory or the field where the specified
apparatusanywhere the specified apparatuses and environmental conditions are can be implemented.
1.5 Units—The values stated in SI units are to be regarded as the standard. The values given in parentheses are not precise
mathematical conversions to inch-pound units. They are close approximate equivalents for the purpose of specifying material
dimensions or quantities that are readily available to avoid excessive fabrication costs of test apparatuses while maintaining
repeatability and reproducibility of the test method results. These values given in parentheses are provided for information only
and are not considered standard.International System of Units (SI Units) and U.S. Customary Units (Imperial Units) are used
throughout this document. They are not mathematical conversions. Rather, they are approximate equivalents in each system of
units to enable use of readily available materials in different countries. This avoids excessive purchasing and fabrication costs. The
differences between the stated dimensions in each system of units are insignificant for the purposes of comparing test method
results, so each system of units is separately considered standard within this test method.
1.6 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility
of the user of this standard to establish appropriate safety safety, health, and healthenvironmental practices and determine the
applicability of regulatory limitations prior to use.
E2826/E2826M − 20
FIG. 2 Three Confined Area Terrain Apparatuses in the Mobility Test Suite with Increasing Complexity; The Continuous Pitch/Roll
Ramps Terrain is Shown on the Left. The Crossing Pitch/Roll Ramps is Shown at the Center. The Symmetric Stepfields Terrain is
Shown on the Right.Top View Showing the Figure-8 Path (forward) Defined by the Barriers
1.7 This international standard was developed in accordance with internationally recognized principles on standardization
established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued
by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
2. Referenced Documents
2.1 ASTM Standards:
E2521 Terminology for Evaluating Response Robot Capabilities
E2592 Practice for Evaluating Response Robot Capabilities: Logistics: Packaging for Urban Search and Rescue Task Force
Equipment Caches
2.2 Additional Documents:Other Standards:
National Response Framework, U.S. Department of Homeland Security
NIST Special Publication 1011–I–2.0 Autonomy Levels for Unmanned Systems (ALFUS) Framework, Volume 1: Terminology,
Version 2.02.04
3. Terminology
3.1 Definitions—The following terms are used in this test method and are defined in Terminology E2521: abstain,administrator
or test administrator,emergency response robot or response robot,fault condition,operator,operator station, remote control,
repetition,robot,teleoperation,test event or event,test form,test sponsor,test suite,testing target or target,testing task or task, and
trial or test trial.
3.2 The following terms are used in this test method and are defined in ALFUS Framework Volume I:3: autonomous,autonomy,
level of autonomy,operator control unit (OCU), and semi-autonomous.
3.3 Definitions:Definitions of Terms Specific to This Standard:
3.1.1 Terminology E2521 lists additional definitions relevant to this test method.
3.1.2 abstain, v—prior to starting a particular test method, the robot manufacturer or designated operator shall choose to enter
the test or abstain. Any abstention shall be granted before the test begins. The test form shall be clearly marked as such, indicating
that the manufacturer acknowledges the omission of the performance data while the test method was available at the test time.
For referenced ASTM standards, visit the ASTM website, www.astm.org, or contact ASTM Customer Service at service@astm.org. For Annual Book of ASTM Standards
volume information, refer to the standard’s Document Summary page on the ASTM website.
Available from Federal Emergency Management Agency (FEMA), P.O. Box 10055, Hyattsville, MD 20782-8055, http://www.fema.gov/emergency/nrf/.
Available from National Institute of Standards and Technology (NIST), 100 Bureau Dr., Stop 1070, Gaithersburg, MD 20899-1070, http://www.nist.gov/customcf/get
_pdf.cfm?pub_id=824705.
3.1.2.1 Discussion—
Abstentions may occur when the robot configuration is neither designed nor equipped to perform the tasks as specified in the test
method. Practice within the test apparatus prior to testing should allow for establishing the applicability of the test method for the
given robot.
E2826/E2826M − 20
3.1.3 administrator, n—person who conducts the test—The administrator shall ensure the readiness of the apparatus, the test
form, and any required measuring devices such as stopwatch and light meter; the administrator shall ensure that the specified or
required environmental conditions are met; the administrator shall notify the operator when the safety belay is available and ensure
that the operator has either decided not to use it or assigned a person to handle it properly; and the administrator shall call the
operator to start and end the test and record the performance data and any notable observations during the test.
3.1.4 emergency response robot, or response robot, n—a robot deployed to perform operational tasks in an emergency response
situation.
3.1.4.1 Discussion—
A response robot is a deployable device intended to perform operational tasks at operational tempos during emergency responses.
It is designed to serve as an extension of the operator for gaining improved remote situational awareness and for projecting her/his
intent through the equipped capabilities. It is designed to reduce risk to the operator while improving effectiveness and efficiency
of the mission. The desired features of a response robot include: rapid deployment; remote operation from an appropriate standoff
distance; mobility in complex environments; sufficient hardening against harsh environments; reliable and field serviceable;
durable or cost effectively disposable, or both; and equipped with operational safeguards.
3.1.5 fault condition, n—during the performance of the task(s) as specified by the test method, a certain condition may occur
that renders the task execution to be failed. Such a condition is called a fault condition. Fault conditions result in a loss of credit
for the partially completed repetition. The test time continues until the operator determines that she/he can not continue and notifies
the administrator. The administrator shall, then, pause the test time and add a time-stamped note on the test form indicating the
reason for the fault condition.
3.1.5.1 Discussion—
Fault conditions include robotic system malfunction, such as de-tracking, and task execution problems, such as excessive deviation
from a specified path or failure to recognize a target.
3.3.1 full-rampquarter-ramp terrain element, n—1.2 by 1.2 m (4 by 4 ft) surface ramp with 15° slope using solid wood support
posts with angle cuts. The material used to build these elements shall beinclined surface of 15° with square dimensions as projected
onto the ground plane equal to ⁄4 strong enough to allow the participating robots to execute the testing tasks.the overall width of
the test lane.
3.1.6.1 Discussion—
The material that is typically used to build these elements, oriented strand board (OSB) is a commonly available construction
material. The frictional characteristics of OSB resemble that of dust covered concrete and other human improved flooring surfaces,
often encountered in emergency responses. Solid wood posts with 10- by 10-cm (4- by 4-in) cross-section dimensions typically
support the ramped surface.
3.1.6.2 Discussion—
Similar elements like this type are used, sometimes mixed and assembled in different configurations, to create various levels of
complexities for robotic functions such as orientation and traction.
3.1.7 half-ramp terrain element, n—0.6-by 1.2-m (2-by 4-ft) surface with the shorter dimension ramped at 15° using solid wood
posts with angle cuts. The material used to build these elements shall be strong enough to allow the participating robots to execute
the allow tasks.
3.1.7.1 Discussion—
See the discussions under full-ramp terrain element.
3.1.8 human-scale, adj—used to indicate that the objects, terrains, or tasks specified in this test method are in a scale consistent
with the environments and structures typically negotiated by humans, although possibly compromised or collapsed enough to limit
human access. Also, that the response robots considered in this context are in a volumetric and weight scale appropriate for
operation within these environments.
3.1.8.1 Discussion—
E2826/E2826M − 20
No precise size and weight ranges are specified for this term. The test apparatus constrains the environment in which the tasks are
performed. Such constraints, in turn, limit the types of robots to be considered applicable to emergency response operations.
3.1.9 operator, n—person who controls the robot to perform the tasks as specified in the test method; she/he shall ensure the
readiness of all the applicable subsystems of the robot; she/he through a designated second shall be responsible for the use of a
safety belay; and she/he shall also determine whether to abstain the test.
3.1.10 operator station, n—apparatus for hosting the operator and her/his operator control unit (OCU, see ALFUS Framework
Volume I: Terminology) to teleoperate (see Terminology E2521) the robot. The operator station shall be positioned in such a
manner so as to insulate the operator from the sights and sounds generated at the test apparatuses.
3.1.11 repetition, n—robot’s completion of the task as specified in the test method and readiness for repeating the same task
when required.
3.1.11.1 Discussion—
In a traversing task, the entire mobility mechanism shall be behind the START point before the traverse and shall pass the END
point to complete a repetition. A test method can specify returning to the START point to complete the task. Multiple repetitions,
performed in the same test condition, may be used to establish the robot performance of a particular test method to a certain degree
of statistical significance as specified by the testing sponsor.
3.1.12 test event or event, n—a set of testing activities that are planned and organized by the test sponsor and to be held at the
designated test site(s).
3.1.13 test form, n—form corresponding to a test method and contains fields for recording the testing results and the associated
information.
3.1.14 test sponsor, n—organization or individual that commissions a particular test event and receives the corresponding test
results.
3.1.15 test suite, n—designed collection of test methods that are used, collectively, to evaluate the performance of a robot’s
particular subsystem or functionality, including mobility, manipulation, sensors, energy/power, communications, human-robot
interaction (HRI), logistics, safety, and aerial or aquatic maneuvering.
3.1.16 testing task, or task, n—a set of activities specified in a test method for testing robots and the operators to perform in
order for the performance to be evaluated according to the corresponding metric(s). A test method may specify multiple tasks.
4. Summary of Test Method
4.1 This test method is performed by a remote operator controlling the robot out of sight and sound of robot within the test
apparatus. The robot follows one of two defined paths in the specified terrain requiring the robot to overcome challenges including
pitch, roll, traction, and turning on uneven surfaces within open or confined spaces.
4.2 The Figure-8 Path (forward) is a continuous forward path through the terrain with alternating left and right turns to avoid
barriers. It can be used to demonstrate terrain traversal over long distances within a relatively small apparatus. The continuous
traverse is shown as the white path (see Fig. 1 and Fig. 2).
4.3 The Zig-Zag Path (forward/reverse) The task for this test method, continuous pitch/roll ramp terrain traversing, is defined
as the robot traversing from the START point along the specified path which ends back at the START point, thus enabling
continuous repetitions. The default path shall be a figure-eight, also known as a continuous “S,” around two pylons installed in
the test course as described in Section is an end-to-end path that requires forward and reverse traversal through the terrain with
alternating left and right turns to avoid barriers. This can be used to demonstrate traversal of the terrain within confined spaces.
The down-range traverse, shown as the white path, is performed in a forward orientation and the up-range traverse, shown as 6.
The START and END points are the same, located beside the first pylon upon enteringthe black path, is performed in reverse (see
Fig. 1 the gate. See and Fig. 3 for an illustration.).
4.4 The robot’s traversing capability of this type of terrain is defined as the robot’s ability to complete the task and the associated
effective speed. Further, the test sponsor can specify the statistical reliability and confidence levels of such a capability and, thus,
dictate the number of successful task performance repetitions that is required. In such a case, the average effective speed shall be
used, instead, as the robot’s capability.robot starts on one side or the other of a lane full of fabricated continuous pitch/roll ramp
terrain at a chosen scale. The robot follows either the figure-8 path (forward) or the zig-zag path (forward/reverse) between the
two barriers. The figure-8 path (forward) repetition is completed when the robot crosses the start/end centerline of the lane without
a fault after approximately following the white path. The zig-zag path (forward/reverse) repetition is completed when the robot
crosses the start/end centerline without a fault after approximately following the white and black paths.
4.5 Potential Faults Include:
4.5.1 Any contact by the robot with the apparatus that requires adjustment or repair to return the apparatus to the initial
condition,
E2826/E2826M − 20
FIG. 3 Mobility: Confined Area Terrains: Continuous Pitch/Roll Ramps Apparatuses (Perspective View)Top View Showing the Zig-Zag
Path (forward/reverse) Defined by the Barriers
4.5.2 Any visual, audible, or physical interaction that assists either the robot or the remote operator, and
4.5.3 Leaving the apparatus during the trial.
4.6 Teleoperation shall be used from the operator station specified by the administrator to test the robots using an OCU provided
by the operator. The operator station shall be positioned and implemented in such a manner so as to insulate the operator from the
sights and sounds generated at the test apparatus.Test trials shall produce enough successful repetitions to demonstrate the
reliability of the system capability or the remote operator proficiency. A complete trial of 10 to 30 repetitions in either one of the
defined paths should take 10 to 30 min to complete. When measuring system capabilities, it is important to allow enough time to
capture a complete trial with an expert operator. When measuring operator proficiency, it is important to limit the time of the trial
so that novice and expert operators are similarly fatigued.
4.7 The operator is allowed to practice before the test. She/he is also allowed to abstain from the test before it is started. Once
the test begins, there shall be no verbal communication between the operator and the administrator regarding the performance
There are three metrics to consider when calculating the results of a test repetition other than instructions on when to start and
notifications of faults and any safety related conditions. The operator shall have the full responsibility to determine whether and
when the robot has completed a repetition and notify the administrator accordingly. However, it is the administrator’s authority to
judge the completeness of the repetition.trial. They should be considered in the following order of importance: completeness score,
reliability, and efficiency. The results from the figure-8 path (forward) and the zig-zag path (forward/reverse) are not comparable
because they measure different capabilities. The results from different scales of test apparatus are also not comparable because they
represent different clearances and distances.
NOTE 2—Practice within the test apparatus could help establish the applicability of the robot for the given test method. It allows the operator to gain
familiarity with the standard apparatus and environmental conditions. It also helps the test administrator to establish the initial apparatus setting for the
test when applicable.
4.5 The test sponsor has the authority to select the size for the specified confined area apparatus. The test sponsor also has the
authority to select the test methods that constitute the test event, to select one or more test site(s) at which the test methods are
implemented, to determine the corresponding statistical reliability and confidence levels of the results for each of the test methods,
and to establish the participation rules including the testing schedules and the test environmental conditions.
5. Significance and Use
5.1 A main purpose of using robots in emergency response operations is to enhance the safety and effectiveness of emergency
responders operating in hazardous or inaccessible environments. The testing results of the candidate robot shall describe, in a
statistically significant way, how reliably the robot is able to traverse the specified types of terrains, thus providing emergency
responders sufficiently high levels of confidence to determine the applicability of the robot.This test method is part of an overall
suite of related test methods that provide repeatable measures of robotic system mobility and remote operator proficiency. This
continuous pitch/roll ramp terrain specifically challenges robotic system locomotion, suspension systems to maintain traction,
rollover tendencies, self-righting in complex terrain (if necessary), chassis shape variability (if available), and remote situational
awareness by the operator. As such, it can be used to represent modest outdoor terrain complexity or indoor debris within confined
areas.
E2826/E2826M − 20
FIG. 4 Mobility: Confined Area Terrains: Continuous Pitch/Roll Ramps Apparatuses (Projection View)Both Paths are Scalable to Repre-
sent Different Environments
5.2 This test method addresses robot performance requirements expressed by emergency responders and representatives from
other interested organizations. The performance data captured within this test method are indicative of the testing robot’s
capabilities. Having available a roster of successfully tested robots with associated performance data to guide procurement and
deployment decisions for emergency responders is consistent with the guideline of “Governments at all levels have a responsibility
to develop detailed, robust, all-hazards response plans” as stated in National Response Framework.The overall size of the terrain
apparatus can vary to provide different constraints depending on the typical obstacle spacing of the intended deployment
environment. For example, the terrain with containment walls can be sized to represent repeatable complexity within bus, train,
or plane aisles; dwellings with hallways and doorways; relatively open parking lots with spaces between cars; or unobstructed
terrains.
5.3 The test apparatuses are scalable to constrain robot maneuverability during task performance for a range of robot sizes in
confined areas associated with emergency response operations. Variants of the apparatus provide minimum lateral clearance of 2.4
m (8 ft) for robots expected to operate around environments such as cluttered city streets, parking lots, and building lobbies;
minimum lateral clearance of 1.2 m (4 ft) for robots expected to operate in and around environments such as large buildings,
stairwells, and urban sidewalks; minimum lateral clearance of 0.6 m (2 ft) for robots expected to operate within environments such
as dwellings and work spaces, buses and airplanes, and semi-collapsed structures; minimum lateral clearance of less than 0.6 m
(2 ft) with a minimum vertical clearance adjustable from 0.6 m (2 ft) to 10 cm (4 in.) for robots expected to deploy through
breeches and operate within sub-human size confined spaces voids in collapsed structures.low cost and easy to fabricate so they
can be widely replicated. The procedure is also simple to conduct. This eases comparisons across various testing locations and
dates to determine best-in-class systems and operators.
5.4 Evaluation—This test method can be used in a controlled environment to measure baseline capabilities. It can also be
embedded into operational training scenarios to measure degradation due to uncontrolled variables in lighting, weather, radio
communications, GPS accuracy, etc.
5.5 Procurement—This test method can be used to identify inherent capability trade-offs in systems, make informed purchasing
decisions, and verify performance during acceptance testing. This aligns requirement specifications and user expectations with
existing capability limits.
5.6 Training—This test method can be used to focus operator training as a repeatable practice task or as an embedded task within
training scenarios. The resulting measures of remote operator proficiency enable tracking of perishable skills over time, along with
comparisons of performance across squads, regions, or national averages.
5.7 Innovation—The standard apparatus is specified to be easily fabricated to facilitate self-evaluation by robot developers and
provide practice tasks for emergency responders that exercise robot actuators, sensors, and operator interfaces. The standard
apparatus can also be used to support operator training and establish operator proficiency.This test method can be used to inspire
technical innovation, demonstrate break-through capabilities, and measure the reliability of systems performing specific tasks
within an overall mission sequence. Combining or sequencing multiple test methods can guide manufacturers toward implementing
the combinations of capabilities necessary to perform essential mission tasks.
5.5 Although the test method was developed first for emergency response robots, it may be applicable to other operational
domains.
6. Apparatus
6.1 The equipment required to perform this test method includes a 15° continuous pitch/roll ramp terrain, barriers to define the
robot path, an optional containment structure, and a timer. The main apparatus dimension to consider is the minimum clearance
width (W) for the robot throughout the specified path (see Fig. 4). The minimum clear width should be chosen to represent typical
obstacle spacings of the intended deployment environment. The minimum clearance width is typically set to 30 cm [1 ft], 60 cm
E2826/E2826M − 20
FIG. 5 Example Top View of a Test Form (Blank)Apparatus Showing Dimensions Scale Proportionally to the Minimum Clearance Width
(W)
[2 ft], 120 cm [4 ft], or 240 cm [8 ft] to efficiently use available construction materials, although other apparatus sizes can be used.
All apparatus dimensions scale proportionally with the minimum clearance width (see Fig. 5). For example, the overall width of
the terrain lane is 2W, and the overall length of the terrain lane is at least 6W. It can be longer for larger robots needing more space
to maneuver around the barriers while staying on the terrain. When choosing a specific minimum clearance width for the apparatus,
note the resulting data is not comparable to other apparatuses with different minimum clearance widths.
6.2 Pitch/Roll Ramp Terrain—The 15° continuous pitch/roll ramp terrain is an array of individual ramps that form peaks and
valleys with no discontinuities. Each ramp is fabricated to fit a square dimension on the ground so it can be rotated in place to form
more difficult terrains. The square ground dimension is set to half the minimum clearance width ( ⁄2 W) so apparatuses at every
scale have a seam between ramps along the centerline of the confined portions of the robot path (see Fig. 6). The ramp surface
can be made of oriented strand board (OSB), plywood, or similar material to provide a relatively consistent low-friction surface.
The supporting structure can be fabricated from lumber posts and OSB panels. Each ramp is fabricated to fit a square dimension
on the ground for interchangeability, so the 15° ramp surface is slightly longer than ⁄2 W in the uphill dimension. The width
remains ⁄2 W. Four lumber posts cut with 15° tops provide support for the ramp surface and connect the three side panels that
provide additional support and enclose the bottom.
6.3 Barriers to Define the Robot Path—The test apparatuses specify three scaled sizes of confined areas fully covered with
full-ramp and half-ramp terrain elements. The three sizes are 7.2 m (24 ft) long by 4.8 m (16 ft), 2.4 m (8 ft), or 1.2 m (4 ft) wide.
Two pylons define the figure-eight path. They are posted at the 2.4- and 4.8-m (8- and 16-ft) distances from either end and along
the centerline between the two sidewalls. The ramps are paired up and connected at the same heights to form ridges or valleys with
no discontinuities, thus they are called “continuous” ramps. The resulting topology causes smoothly recurring orientation
complexities for robots. The terrain is surrounded with a 1.2 m (4 ft) tall wall. A gate opens in the front to allow robot entry. See
barriers placed within the terrain must provide visual guidance Figs. 3 and 4. Each repetition for the figure eight path is nominally
considered to be 15 m long. Sectionfor the remote robot operator to correctly traverse the defined figure-8 path (forward) or zig-zag
path (forward/reverse). The barrier can be made from any solid 5.3 specifies the applicability of each scale of the apparatuses.or
porous material that provides visual guidance. They should be sturdy and easily repaired or replaced after contact with the robot.
The barrier’s overall thickness shall remain less than 5 % of the minimum clearance width and the length shall equal W.
NOTE 3—The material that is typically used to build the test apparatuses, OSB, is a commonly available construction material. The frictional
characteristics of OSB resemble that of dust covered concrete and other improved flooring surfaces often encountered in emergency responses.
6.4 Containment Structure—Various test conditions such The array of individual pitch/roll ramps need to be contained so they
do not move relative to one another (see Fig. 8as apparatus surface types and conditions, including wetness and friction levels,
temperature, types of lighting, smoke, humidity, and rain shall be facilitated when the test sponsor requires. For example, for a test
run in the dark environment, a light meter shall be used to read 0.1 lux or less. The darkness shall be re-measured when the lighting
condition might have changed. The actual readings of these conditions should be recorded on the test form.). The minimum
containment is an underlayment with an affixed lumber border. Walls can be fabricated to contain the robot as well as the terrain.
This provides an extra level of difficulty for the robot. It can also provide a safety barrier for nearby personnel within a test facility.
The fabricated wood walls are typically supported with arches over top. Shipping containers can also enclose test methods and turn
a parking lot into a test facility. Apparatuses with minimum clearance width W = 120 cm [4 ft] can be slightly undersized to fit
into a standard shipping container which has an interior width less than 2400 cm [8 ft]. The container walls should be lined with
wood panels to cover the corrugated steel and have enough thickness to fill any gaps between the wall and the terrain.
NOTE 4—The testing apparatus can be implemented in an International Standards Organization (ISO) specified standard shipping container in which
some of the test conditions can be furnished. To achieve the specified darkness, first turn off all the lighting sources inside and entirely cover the entrance
with light-blocking drapes. The darkness is specified as 0.1 lux due to the implementation cost concerns for the apparatuses and due to the fact that robotic
cameras are less sensitive than human eyes, such that any darkness below 0.1 lux would not make a difference in the cameras’ functioning. It is recognized
E2826/E2826M − 20
FIG. 6 Example of a Test Form (Filled in with Illustrative Data)Details of a Pitch/Roll Ramp
FIG. 7 Side View Showing Various Barriers
FIG. 8 Various Containment Structures
that the environments in real applications may be darker than the specified test condition.
6.5 Other Devices—A stopwatch shall be provided timer is used to measure the timing performance.elapsed time of the trial.
It provides a deterministic indication of trial start and end times to minimize uncertainty. It can count-up or count-down but should
E2826/E2826M − 20
have a settable duration in minutes. A stopwatch can also be used. A light meter is necessary if testing in lighted and dark
environments. A lighted environment is considered >150 lx and dark environment is considered <0.1 lx.
7. Hazards
7.1 Besides 1.4 that addresses the human safety and health concerns, users of the standard shall also address the equipment
preservation concerns and human-robot coexistence concerns.
NOTE 5—The environmental conditions, such as high or low temperatures, excessive moisture, and rough terrains can be stressful to the humans, can
damage the robotic components, or can cause unexpected robotic motions.
7. Hazards
7.1 Functional emergency stop systems are essential for safe remote or autonomous robot operation. The emergency stop on the
operator control unit shall be clearly marked and accessible. The emergency stop on the robot chassis, if available, should also be
marked. All personnel involved in testing shall familiarize themselves with the locations of all emergency stops prior to conducting
trials.
7.2 Emergency stop systems shall be engaged prior to approaching a remotely operated robot. Constant communication is
essential between the robot and the operator until the robot is safely within the test apparatus and people are either outside the
apparatus or at a safe distance. The remote operator may not be aware that someone is interacting with the robot when they start
to drive, actuate a manipulator, or move the robot in some other way. Avoid standing directly in front of the robot, behind the robot,
or within reach of the manipulator arm unless the robot is completely deactivated.
7.3 Safety equipment such as a belay shall be used from a safe distance to prevent robot damage if necessary. Intervention by
hand to try to stop a robot from falling or flipping over is to be prohibited. The belay shall be required for this. Any interaction
with the robot, including tightening the belay to save the robot, is considered a fault for scoring purposes.
7.4 Test apparatuses that are intended to challenge robot mobility can be complex and unstable for humans. Proper footwear
and other personal protective equipment shall be worn to mitigate risk. Caution is required when attending to a robot or carrying
equipment within the apparatus.
8. Calibration and Standardization
8.1 The robot configuration as tested shall be described in detail on the test form, including all subsystems and components and
their respective features and functionalities. The configuration shall be subjected to all the test suites, as defined in 3.1.15, as
appropriate. Any variation in the configuration shall cause the resulting robot variant to be retested across all the test suites to
provide a consistent and comprehensive representation of the performance. Practice E2592 shall be used to record the robotic
configuration.
8.2 Once a robot begins a test, by starting executing the task as specified in 4.1, the robot shall be teleoperated to perform the
task for the specified number of repetitions through completion without leaving the apparatus. During the process, the robot shall
not be allowed to have the energy/power source replenished nor shall the robot be allowed any human physical intervention,
including adjustment, maintenance, or repair. Any such actions shall be considered a fault condition.
8.3 The metric for this test method is the completeness of the prescribed path successfully traversed for the specified number
of continuous repetitions.
8.4 In addition, the elapsed time for successfully traversing the path, or effective speed in meters per minute, is a performance
proficiency index reflecting the combination of the robot’s capability and efficiency, the OCU’s ease of use, and the operator’s skill
level. Therefore, this temporal aspect is a part of the test and the results shall be recorded on the test form.
NOTE 6—The term “effective” is used because the speed is calculated based on the designed length of the path and not on the actual path of the
traverses, which can deviate from the designed path.
8.5 Although the metric is based on teleoperation, autonomous behaviors are allowed as long as the testing procedure is
followed, with the associated effects reflected in the testing scores. See ALFUS Framework Volume I: Terminology for the
definition of autonomy.
8.6 The test sponsor has the authority to specify the lighting condition and other environmental variables, which can affect the
test results. All environmental settings shall be noted on the test form.
8.7 A robot’s reliability (R) of performing the specified task at a particular apparatus setting and the associated confidence (C)
shall be established. The required R and C values dictate the required number of successful repetitions and the allowed number
of failures during the test. With a given set of the R and C values, more successes will be needed when more failures are allowed.
A test sponsor has the authority to specify the R and C values for her/his testing purposes, otherwise she/he can elect to use the
default values for this standard. The factors to be considered in determining the values are mission requirements, consistency with
the operating environments, ease of performing the required number of repetitions, and testing costs such as time and personnel.
To meet the statistical significance established by the standards committee, which is 80 % reliability (probability of success) with
E2826/E2826M − 20
85 % confidence at any given setting of a test apparatus, the number of failures (incomplete repetitions or the occurrence of the
fault conditions) in the specified set of repetitions shall be no more than the following:
zero failures in 10 repetitions
one failure in 20 repetitions
three failures in 30 repetitions
four failures in 40 repetitions
six failures in 50 repetitions
eight failures in 60 repetitions
NOTE 7—The two-failure and five-failure situations are omitted in order to have the total repetition numbers increment in sets of 10 consistently to
ease test administration.
8.7.1 Additional repetition requirements can be calculated, if a test sponsor requires, by referring to general statistical analysis
methods.
8. Procedure
8.1 Identify the Robot Configuration— For data traceability and organization purposes, the administrator shall obtain and record
the pre-test information first. A set of specified fault conditions shall be followed during the test.The robotic system configuration
being tested shall be identified and uniquely named (for example, make, model, configuration), including all subsystems and
components with their respective features and functionalities. The configuration of the robotic system should be representative of
a configuration that will be used in its intended application. A given robotic system may have several different configurations. Any
number of configurations can be subjected to testing. The system configuration shall remain the same for all relevant tests to enable
direct comparison of performance and to identify trade-offs between different configurations. In general, robotic system
configurations shall maintain their overall cubic volume, weight, and center of gravity, as well as major sub-systems such as tracks,
wheels, legs, manipulator, radio comms, tether, operator control unit, etc. More subtle changes within the system or software are
harder to track and may typically be disregarded. If the robot configuration is changed during a trial it is considered invalid.
Documentation should include detailed photographs of all of the above as well as videos of routine maintenance tasks such as a
...