ASTM E3124-17

This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the
Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
Designation: E3124 − 17
Standard Test Method for
Measuring System Latency Performance of Optical Tracking
Systems that Measure Six Degrees of Freedom (6DOF)
Pose
This standard is issued under the fixed designation E3124; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
1. Scope 2. Referenced Documents
1.1 Purpose—This test method presents metrics and a pro- 2.1 ASTM Standards:
cedure for measuring, analyzing, and reporting the system E177 Practice for Use of the Terms Precision and Bias in
latency of an optical tracking system (OTS) that computes the ASTM Test Methods
pose of a rigid object. E2655 Guide for Reporting Uncertainty of Test Results and
Use of the Term Measurement Uncertainty inASTM Test
1.2 Usage—System vendors may use this test method to
Methods
determine or validate the system latency in their tracking
E2919 Test Method for Evaluating the Performance of
systems. This test method provides a uniform way to measure
Systems that Measure Static, Six Degrees of Freedom
and report the system latency along with the uncertainty in the
(6DOF), Pose
systemlatency.Systemusersmayusethistestmethodtoverify
E3064 Test Method for Evaluating the Performance of
that the system latency performance is within the user’s
Optical Tracking Systems that Measure Six Degrees of
specific requirements and within the system’s rated perfor-
Freedom (6DOF) Pose
mance.
2.2 ASME Standard:
1.3 This standard does not measure the display latency of
ASME B89.4.19 Performance Evaluation of Laser-Based
graphical representations of the tracked objects. Display la-
Spherical Coordinate Measurement Systems
tency is external to the optical tracking system.
3. Terminology
1.4 Test Location—The procedures defined in this test
method shall be performed in an environment conforming to
3.1 Definitions:
the manufacturer’s rated conditions.
3.1.1 degrees of freedom, DOF, n—any of the minimum
number of translation or rotation components required to
1.5 The values stated in SI units are to be regarded as
specify completely the pose of a rigid object. E2919
standard. No other units of measurement are included in this
standard.
3.1.1.1 Discussion—
1.6 This standard does not purport to address all of the (1) In a 3D space, a rigid object’s pose can be minimally
safety concerns, if any, associated with its use. It is the represented by 6DOF, three translations and three rotations.
responsibility of the user of this standard to establish appro- (2) The term “degrees of freedom” is also used with regard
priate safety, health, and environmental practices and deter- to statistical testing. It will be clear from the context in which
mine the applicability of regulatory limitations prior to use.
it is used whether the term relates to a statistical test or the
1.7 This international standard was developed in accor- rotation/translation aspect of the object.
dance with internationally recognized principles on standard- 3.1.2 frame rate, n—frequency at which a camera acquires
ization established in the Decision on Principles for the consecutive images.
Development of International Standards, Guides and Recom-
3.1.3 integration time, n—the length of time when the
mendations issued by the World Trade Organization Technical
digital sensor inside a camera collects light.
Barriers to Trade (TBT) Committee.
For referenced ASTM standards, visit the ASTM website, www.astm.org, or
contact ASTM Customer Service at service@astm.org. For Annual Book of ASTM
This test method is under the jurisdiction of ASTM Committee E57 on 3D Standards volume information, refer to the standard’s Document Summary page on
ImagingSystemsandisthedirectresponsibilityofSubcommitteeE57.50onOptical the ASTM website.
Tracking Systems. Available from American Society of Mechanical Engineers (ASME), ASME
Current edition approved Oct. 1, 2017. Published December 2017. DOI: International Headquarters, Two Park Ave., New York, NY 10016-5990, http://
10.1520/E3124-17. www.asme.org.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
E3124 − 17
3.1.3.1 Discussion—In some systems integration time is quisition times. As indicated in Section 9, vendors should
also called exposure time. report the operating parameters of the system including
system-wide integration periods and camera frame rates.
3.1.4 optical image event, n—the instant in time when the
3.1.13 temporal pose error—the pose error in time domain.
OTS registers an event.
3.1.4.1 Discussion—In most systems, it is defined either by 3.1.14 tracking system, n—a system that is used for mea-
the beginning or the center of the image integration time. suring the pose of moving objects and supplies the data as a
timely ordered sequence. E3064
3.1.5 optical tracking system, n—atrackingsystemthatuses
measurements obtained from camera images. E3064 3.1.15 uncertainty, n—an indication of the magnitude of
error associated with a value that takes into account both
3.1.6 physical event, n—apointintimecorrespondingtothe
systematic errors and random errors associated with the mea-
physical motion of a target object tracked by the OTS in the
surement or test process. E2655
test space (see Fig. 1).
3.1.7 physical event latency, n—time between an actual
4. Summary of Test Method
occurrence and OTS report of the corresponding occurrence.
4.1 This test method provides a set of statistically-based
3.1.8 pose, n—a 6DOF vector whose components represent
performance metrics and a test procedure to quantitatively
the position and orientation of a rigid object with respect to a
measure the system latency of an optical tracking system.
coordinate frame. E2919
4.2 Specifically, the test procedure measures the difference
3.1.9 precision, n—the closeness of agreement between
between the time instant when the motion of a test apparatus
independent test results obtained under stipulated conditions.
causes an electric circuit to be closed and the instant when the
E177
opticaltrackingsystemhasdetectedtheevent.Thetimeinstant
3.1.10 rated conditions, n—manufacturer-specified limits
when the electric circuit is closed is measured by a low-latency
on environmental, utility, and other conditions within which
data acquisition system which is synchronized with the optical
the manufacturer’s performance specifications are guaranteed
tracking system. The performance metrics include the mean,
at the time of installation of the instrument. ASME B89.4.19
the standard deviation, the maximum, the minimum, and
certain percentiles of latency measurements.
3.1.11 standard uncertainty, n—uncertainty reported as the
standard deviation of the estimated value of the quantity
5. Significance and Use
subject to measurement. E2655
5.1 Optical tracking systems are used in a wide range of
3.1.12 system latency, n—the elapsed time between the
fields including: video games, film, neuroscience,
optical image event and the instant in time when the client
biomechanics, flight/medical/industrial training, simulation,
receives the 6DOF pose information corresponding to that
robotics, and automotive applications.
event from the optical tracking system (OTS).
5.2 This standard provides a common set of metrics and a
3.1.12.1 Discussion—In Fig. 1, the optical image event time
test procedure for evaluating the performance of optical
is marked by the letter “B” and the time of receipt of the pose
tracking systems and may help to drive improvements and
data by a client system is marked as “D”. OTS computation
innovations in optical tracking systems.
time is the time for the completion of the computation of the
5.3 Potential users often have difficulty comparing optical
pose data by the OTS. OTS communication latency is the time
tracking systems due to the lack of standard performance
between the completion of the computation of the pose data
metricsandtestmethods,andmustthereforerelyonthevendor
and the receipt of the data by the client.
claims regarding the system’s performance, capabilities, and
3.1.12.2 Discussion—Certain optical tracking systems will
observe the optical image event some amount of time after the
physical event “A” happens. In the diagram of Fig. 1, OTS 4
“Motion Capture Software Developers in the US: Market Research Report,”
computation time incorporates the vendor-specific image ac- IBIS World, 2014.
FIG. 1 OTS Latency
E3124 − 17
suitability for a particular application. This standard makes it illustrative design is given in Appendix X2. Trackable ele-
possible for a user to assess and compare the performance of ments such as markers should be attached to the device
opticaltrackingsystems,andallowstheusertodetermineifthe according to the vendor requirements (for example, minimum
measured performance results are within the specifications number of markers and marker locations) so that it can be
with regard to the application requirements. accurately tracked as a rigid body.
6.3 Other Materials—This test method also requires a
6. Apparatus
computer (that is, the client system), a solid surface, two wires,
6.1 Data Acquisition Device—Forbestresults,alowlatency
and a conductive plate (Fig. 2).Additionally, depending on the
Data Acquisition Device (DAQ) should be used. Although
specifications provided by the DAQ manufacturer, a pull-down
there are no requirements on the DAQ, a device which is
resistor (that is, resistor to ground connection) or a pull-up
integrated into the computer via a PCI Express port has shown
resistor (that is, resistor to power supply connection) may be
acceptable performance. Other DAQ devices may be used, but
necessary. The resistance of the pull-down/pull-up resistor
they may incur additional latency and uncertainty to the tests,
should be defined according to the DAQ manufacturer recom-
which may be incorrectly construed as OTS system latency.
mendation (10,000 Ohm is a typical value). To ensure repeat-
While PCI Express devices tend to show sub-microsecond
able contact conditions, the conductive plate should not move
latencies,USB3.0and2.0devices,forexample,havelatencies
relative to the hard surface on which it is set even under the
between tens of microseconds to approximately one hundred
action of the hammer striking the conductive plate.
microseconds nad Gigabit Ethernet devices can incur addi-
tional latencies on the order of one millisecond. The user
7. Measurement and Test Procedure
should also make sure that the operating system does not
7.1 Introduction—This section describes the basic proce-
introduce significant delays into the measurements and that
dure for measuring the latency of an optical tracking system.
there are no processes running in the background during the
Latency measurements shall be carried out with a single
testprocedure.Ingeneral,theusershouldminimizeinstrumen-
trackable object without other objects in the background.
tation related latencies. Standalone measurement devices may
Optinally, vendors may choose to carry out additional tests
also be used to avoid these issues as long as they can produce
with multiple trackable objects, in which case object count
timestamps which are synchronized with those generated by
shall be included in the report.
the OTS.
7.1.1 Setup—Attach a wire to the head of the hammer
6.2 Hammer Device—A device such as a hammer or pen-
device and another wire to a conductive plate. Connect the
dulum shall be designed in such a way that it can be released
wires from the hammer and the conductive plate to a DAQ
and free fall onto a conductive plate (for example, a copper
device connected directly into the computer. In the optical
plate or other low-latency devices that can form electrical
tracking system software make the hammer device a trackable
contact such as a contact switch). Although there are no strict
object and start streaming its posi
...