ISO/IEC 21134:2026

International
Standard
ISO/IEC 21134
First edition
Information technology —
2026-01
Computer graphics, image
processing and environmental data
representation — Benchmarking
of integrated indoor localization
and tracking methods using dead
reckoning
Reference number
© ISO/IEC 2026
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on
the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below
or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
© ISO/IEC 2026 – All rights reserved
ii
Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
3.1 Terms and definitions .1
3.2 Acronyms and abbreviated terms .3
4 Overview of the framework . 4
5 Benchmarking processes . . 4
5.1 Overview .4
5.2 Benchmarking flow .5
5.3 Stakeholders .5
5.4 Benchmarking types .5
6 Benchmark metrics . 6
6.1 Overview .6
6.2 Indicators related to position .7
6.3 Indicators related to series of positional estimation .10
6.4 Indicators related to orientation .10
6.5 Indicators related to relative relationship .10
6.6 Indicator related to integrated localization .11
6.7 Indicators related to practicability of localization system .11
6.8 Indicators related to data used for the benchmarking .11
7 Master set for benchmarking .11
7.1 Overview .11
7.2 Dataset for off-site benchmarking . 12
8 Conformance .12
Annex A (informative) Benchmarking activities . 14
Annex B (informative) Usage examples of conformation checklists .23
Annex C (informative) Difference between this document and other standards .36
Bibliography .37

© ISO/IEC 2026 – All rights reserved
iii
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical activity.
ISO and IEC technical committees collaborate in fields of mutual interest. Other international organizations,
governmental and non-governmental, in liaison with ISO and IEC, also take part in the work.
The procedures used to develop this document and those intended for its further maintenance are described
in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for the different types
of document should be noted. This document was drafted in accordance with the editorial rules of the ISO/
IEC Directives, Part 2 (see www.iso.org/directives or www.iec.ch/members_experts/refdocs).
ISO and IEC draw attention to the possibility that the implementation of this document may involve the
use of (a) patent(s). ISO and IEC take no position concerning the evidence, validity or applicability of any
claimed patent rights in respect thereof. As of the date of publication of this document, ISO and IEC had not
received notice of (a) patent(s) which may be required to implement this document. However, implementers
are cautioned that this may not represent the latest information, which may be obtained from the patent
database available at www.iso.org/patents and https://patents.iec.ch. ISO and IEC shall not be held
responsible for identifying any or all such patent rights.
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and expressions
related to conformity assessment, as well as information about ISO's adherence to the World Trade
Organization (WTO) principles in the Technical Barriers to Trade (TBT) see www.iso.org/iso/foreword.html.
In the IEC, see www.iec.ch/understanding-standards.
This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 24, Computer graphics, image processing and environmental data representation.
Any feedback or questions on this document should be directed to the user’s national standards
body. A complete listing of these bodies can be found at www.iso.org/members.html and
www.iec.ch/national-committees.

© ISO/IEC 2026 – All rights reserved
iv
Introduction
In the development of location-based service and application such as navigation, traffic line analysis, one of
the most important parts is the localization and tracking method (LTM) used for estimating the position of
the target people or objects. In contrast to the outdoor, global navigation satellite systems cannot be used
in indoor environment because the signals from the satellite cannot be reached. In order to provide the
solution for localization and tracking in indoor environment, the research and development on integrated
indoor LTMs is flourishing and many new algorithms are proposed every year. Especially, dead reckoning
technologies (xDR) such as pedestrian dead reckoning (PDR) can contribute integrated indoor localization,
because of their capability of relative tracking without any infrastructure in the target environment.
Therefore, this document intends to foster objective evaluation and comparison of diverse LTMs, especially
for targeting integrated LTMs using dead reckoning.
The target audience of this document includes stakeholders of benchmarking activities. The following are
examples of how this document can be used directly or indirectly:
— by a benchmarking service provider, a benchmark provider or a benchmarking competition organizer
who wishes to align their benchmarking activities including self-benchmarking and open or closed
competitions to be consistent with this document;
— by a technology developer or supplier who wishes to estimate and evaluate the performance of an indoor
localization and tracking methods appropriately with a benchmarking service provider, a benchmark
provider or a benchmarking competition organizer who aligns their benchmarking activities to be
consistent with this document; or
— by a technology user who wishes to obtain benchmarking results based on a benchmarking activity,
which is consistent with this document, or to compare the existing indoor localization and tracking
methods in terms of their performance.
The main targets of the benchmarking framework are integrated indoor LTMs using dead reckoning, which
are assumed to be running on the mobile devices such as smartphones.
a) Master sets and test environments:
The framework defines reference master sets and test environments, where integrated indoor
localization methods using dead reckoning is assumed to be utilized. The reference master sets include
dataset designed for evaluating individual contribution of dead reckoning method.
b) Benchmark metrics:
The framework defines benchmark metrics consisting of indicators for dead reckoning and other
indicators evaluation practical performance of indoor localization methods.
c) Benchmarking process:
The framework defines benchmarking process which can evaluate each elemental performance of the
contents of the integrated indoor localization methods and generalization performance.
d) Conformance.
The framework defines conformance check list which can clarify how each benchmarking activity
conforms to the benchmarking framework.

© ISO/IEC 2026 – All rights reserved
v
International Standard ISO/IEC 21134:2026(en)
Information technology — Computer graphics, image
processing and environmental data representation —
Benchmarking of integrated indoor localization and tracking
methods using dead reckoning
1 Scope
This document specifies the reference framework for the benchmarking of integrated indoor localization
and tracking methods (LTMs) using dead reckoning in areas of:
— master sets and test environments;
— benchmark metrics;
— benchmarking process;
— conformance.
2 Normative references
There are no normative references in this document.
3 Terms and definitions
For the purposes of this document, the following terms and definitions apply.
ISO and IEC maintain terminology databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at https:// www .electropedia .org/
3.1 Terms and definitions
3.1.1
localization and tracking method
method for estimating location of the target and continuously tracking the movement of the target over time
3.1.2
pedestrian dead reckoning
method for continuously estimating pedestrian’s relative change of position and heading direction from a
known position and heading direction by only using user mounted sensors such as inertial measurement
unit, compass, barometer and other type of sensors which can be mounted on the user
3.1.3
error accumulation gradient
speed of error accumulation caused by the nature of relative tracking and includes variations such as
position error accumulation gradient and orientation error accumulation gradient
3.1.4
position error accumulation gradient
speed of positional error accumulation caused by the nature of relative tracking

© ISO/IEC 2026 – All rights reserved
3.1.5
orientation error accumulation gradient
speed of orientational error accumulation caused by the nature of relative tracking
3.1.6
absolute localization
localization method that estimates absolute position and posture of the target
3.1.7
relative localization
localization method that estimates relative change of position and posture of the target
3.1.8
absolute localization inapplicable period
1)
period in dataset when data for the absolute localization such as records of Bluetooth® low energy (BLE)
signals are deleted
3.1.9
absolute localization applicable period
period in dataset when data for the absolute localization are available (not deleted)
3.1.10
correction point
point whose ground-truth positions is provided by benchmark organizers for positional correction
3.1.11
evaluation point
point whose position is hidden from developers, and used for evaluation by competition organizers or
benchmarking service providers
3.1.12
moving velocity check
negative check elements regarding moving velocity
3.1.13
obstacle avoidance check
negative check elements regarding objective avoidance
3.1.14
circular accuracy
accuracy evaluation elements evaluated by the deviation of the error distribution in 2D (xy) error space
3.1.15
un-walkable area
area defined in floor map as areas where tracking targets cannot walk into
3.1.16
velocity error
error of velocity compared with correct velocity
3.1.17
scoring dataset
dataset provided for actual benchmarking without ground-truth (GT) data
Note 1 to entry: Participants are asked to submit the result of LTMs by processing the scoring dataset.
1) Bluetooth is a tradename of a product supplied by Bluetooth Special Interest Group. This information is given for the
convenience of users of this document and does not constitute an endorsement by ISO or IEC of the product named.

© ISO/IEC 2026 – All rights reserved
3.2 Acronyms and abbreviated terms
AC-OEAG angular change normalized orientation error accumulation gradient
AC-PEAG angular change normalized position error accumulation gradient
AL absolute localization
ALAP absolute localization applicable period
ALIP absolute localization inapplicable period
AP access point
API application programming interface
BLE Bluetooth low energy
CA circular accuracy
CE circular error
DE distance error
D-OEAG distance normalized orientation error accumulation gradient
DOF degree of freedom
D-PEAG distance normalized position error accumulation gradient
EAG error accumulation gradient
GT ground truth
IMU inertial measurement unit
IPIN international conference on indoor positioning and indoor navigation
LiDAR light detection and ranging (or laser imaging detection and ranging)
LTM localization and tracking method
NTP network time protocol
OEAG orientation error accumulation gradient
PDR pedestrian dead reckoning
PEAG position error accumulation gradient
PIEM path independent evaluation metric
RSSI received signal strength indicator
T-OEAG time normalized orientation error accumulation gradient
T-PEAG time normalized position error accumulation gradient
VE velocity error
vSRT vision-based spatial registration and tracking

© ISO/IEC 2026 – All rights reserved
WMS warehouse management system
xDR (generic expression of) dead reckoning
4 Overview of the framework
This clause outlines the reference framework of benchmarking of integrated indoor LTMs using dead
reckoning, the details of which are described in Clauses 5, 6, and 7. As shown in Figure 1, the reference
framework is composed of three core components.
— Benchmarking processes, which include how to produce benchmarking outcomes such as benchmarking
results, benchmark surveys and benchmarking instruments with benchmark indicators and trial sets
and how to share benchmarking outcomes.
— Benchmark indicators, which quantify the performance of LTMs by taking into account not only
positioning accuracy, but also various and practical performance required for LTMs in realistic situation.
— Master set, which is composed of datasets and physical testing fields to provide each benchmarking
attempt with the same conditions.
Figure 1 — Core components of the benchmarking framework
The above three components are identified and defined in accordance with grass-roots activities for
standardizing benchmarking schemes and for conducting on-site and off-site comparison of LTMs which are
often held as contests and are introduced in Annex A.
5 Benchmarking processes
5.1 Overview
This clause outlines benchmarking processes and related components necessary to produce and share
benchmarking outcomes. Figure 2 illustrates the basic benchmarking process flow. The benchmarking
process for indoor LTMs is a derivative of the benchmarking process for vSRT standardized in ISO/IEC 18520.

© ISO/IEC 2026 – All rights reserved
Figure 2 — Basic benchmarking flow
5.2 Benchmarking flow
Just as the benchmarking flow for vSRT specified in ISO/IEC 18520, the benchmarking flow for indoor
LTMs consists of process, target, input, output, outcome, and organized storage as shown in Figure 2. The
differences from ISO/IEC 18520 are target system and master sets. Target of the benchmarking for indoor
LTMs is location-based system which its location is provided by indoor LTMs.
Master set of the benchmarking for indoor LTMs consists of either datasets or physical fields, or both. Datasets
are created for benchmarking using recorded data. Datasets contained in the master sets are extracted from
the benchmarking repositories. The GT data contained in the dataset, if the dataset is provided for training,
learning and self-evaluation for technology developer. The GT data is hidden if the dataset is provided for
evaluation by benchmarking provider. Supplemental information is informative information for clarifying
the situation for the benchmarking activity. Physical fields are provided by the benchmarking organizer for
benchmarking the targets in the same physical situation.
5.3 Stakeholders
Just as the stakeholders of benchmarking for vision-based spatial registration and tracking (vSRT)
standardized in ISO/IEC 18520, stakeholders of benchmarking indoor LTMs are categorized into benchmark
providers and benchmark service providers, quality verifiers, technology developers, technology suppliers
and technology users. The target technologies for technology developers, technology suppliers and
technology users are indoor LTMs or location-based systems.
5.4 Benchmarking types
Benchmarking activity can be classified according to the temporal and spatial situations as shown in Table 1.
Temporal classification and spatial classification are merged for specifying the situations. Competitions

© ISO/IEC 2026 – All rights reserved
in international conference on indoor position and indoor localization (IPIN) started to categorize their
[5]
competition tracks in this way.
Classifications from a temporal perspective include on-line and off-line:
— On-line benchmarking, which is benchmarking under the real-time constraint. The measured data or
provided data should be processed in real time. The estimated results should be submitted in real-time.
— Off-line benchmarking, which is benchmarking under no real-time constraint. The participants of the
benchmarking can grasp whole data when process the data and can apply global optimization.
Classifications from a spatial perspective include on-site and off-site:
— On-site benchmarking, which is benchmarking whose participants of the benchmarking run their method
at the actual site.
— Off-site benchmarking, which is benchmarking whose participants of the benchmarking do not run their
method at the actual site. The benchmark organizer goes to the target site and records the data required
for the target method on behalf of the participants of the benchmarking.
Table 1 — Benchmarking types
Temporal classification
Benchmarking types
On-line Off-line
“On-site, On-line” “On-site, Off-line”
On-site
benchmarking benchmarking
Spatial classification
“Off-site, On-line” “Off-site, Off-line”
Off-site
benchmarking benchmarking
6 Benchmark metrics
6.1 Overview
This clause outlines seven categories of benchmark metrics (position related indicators, orientation related
indicator, indicators related to series of positional estimation, indicator related to relative relationship,
indicator related to integrated localization, supplemental indicators related to practicability of localization
system, supplemental indicators related to data used for the benchmarking), which should be utilized for
fair comparison of indoor LTMs. Table 2 shows representative benchmark metrics and indicators for indoor
localization and tracking methods using dead reckoning.
Table 2 — Metrics and indicators for benchmarking
Categories of metrics and indica- Metrics and indicators Attributes or source of indicators
tors
Indicators related to position — Position error accumulation — PIEM
gradient (PEAG, time/distance/
— EAG
angular change normalized PEAG)
— Circular error — ISO/IEC 18305
— Circular accuracy (in world/local
coordinate systems)
— Area detection performance
— Validity of trajectory — Negative checks
Indicators related to series of posi- — Velocity error (VE) — PIEM
tional estimation
— Distance error
— Moving velocity check — Negative checks

© ISO/IEC 2026 – All rights reserved
TTabablele 2 2 ((ccoonnttiinnueuedd))
Categories of metrics and indica- Metrics and indicators Attributes or source of indicators
tors
Indicators related to orientation — Orientation error accumulation — PIEM
gradient (OEAG, time/distance/
— EAG
angular change normalized OEAG)
Indicators related to relative rela- — Relative distance accuracy — ISO/IEC 18305
tionship
— Relative pose accuracy
Indicator related to integrated local- — Difference between the
ization performances in ALIP and ALAP
Indicators related to practicability of Set-up time, cost, easiness for users, — ISO/IEC 18305
localization system availability, through-put, latency
— ISO/IEC 18520 (thorough-put,
latency)
— Negative check (availability)
Indicators related to data used for Number of datasets, variety on proper- — ISO/IEC 18520
the benchmarking ties of datasets
Table 2 also contains a column of attributes or source of indicator. This column indicates supplemental
attributes or its source to which the indicators belong. Path independent evaluation metric (PIEM) and
negative checks are possible attributes shown in this column. Other standards referenced are listed in this
column as a source of the indicator.
[4]
— Path independent evaluation metric (PIEM), which is an attribute of the metric that is designed for
consistently evaluating LTMs no matter where the path of the target is passing. If the paths vary in terms
of length and complexity, PIEM compensates them in evaluation.
— Error accumulation gradient (EAG) is an example of the concept of PIEM. EAG evaluates speed of
error accumulation caused by the nature of relative tracking. EAG has multiple derivatives according
to the type of error.
[3]
— Negative checks check results of estimation based on minimum standards that shall be fulfilled for an
ideal method. These metrics can be evaluated without GT.
6.2 Indicators related to position
This subclause presents indicators related to position, which is a type of indicators related to the error of
position estimation by LTMs. If the benchmarking organizer or datasets have GT of position, this category of
metric can be evaluated. This category of the indicators consists of metrics and indicators as follows.
— Position error accumulation gradient (PEAG) is speed of positional error accumulation caused by relative
tracking as shown in Figure 3. This is an example of the EAG and PIEM. This evaluation metric can be
calculated if correct position of the target is available and modification of the sensor data in datasets are
made for defining absolute localization inapplicable period (ALIP) and absolute localization applicable
period (ALAP). In ALIP, absolute localization (AL) cannot be applied for estimating pure contribution of
the xDR. For example, BLE signals, a major source for AL, are intentionally deleted in ALIP as shown in
Figure 4. There are multiple derivatives of PEAG according to the difference of normalization.
— Time normalized position error accumulation gradient (T-PEAG).
— Distance normalized position error accumulation gradient (D-PEAG).
— Angular change normalized position error accumulation gradient (AC-PEAG).

© ISO/IEC 2026 – All rights reserved
Key
close to correction points (CPs)
far from correction points
un-walkable area
true trajectory
estimated trajectory
evaluation points
corresponding points on estimated
trajectory correction points (CP)
circular error
Figure 3 — PEAGs
© ISO/IEC 2026 – All rights reserved
Key
correction points are provided
evaluation points
Figure 4 — ALIP and ALAP
— Circular error (CE), which is absolute two-dimensional positional error compared to GT at evaluation
points. This metrics is defined by ISO/IEC 18305. Example indicator is CE50 which stands for 50
percentiles of the CE.
— Circular accuracy (CA), which is bias of error distribution in 2D error space as shown in Figure 5. CE
cannot separately evaluate accuracy and precision. CA is an evaluation metrics for evaluating accuracy.
There are multiple derivatives of CA according to the difference of the coordinate systems where errors
are evaluated.
— CA is calculated by deviation of the error distribution in the world coordinate system.
W
— CA is calculated by deviation of the error distribution in the local coordinate system. Errors
L
of the position are calculated every reference point in the world coordinate. The coordinate
system of errors is transformed to the local coordinate system by using orientation of the GT.
Figure 5 — CA and CA
W L
— Area detection performance, which is the error of area detection of the LTMs. Assuming the simplest case
in which the LTM outputs time series of stayed area names for each timestamp and the corresponding
GT is available, this indicator is measured as precision, recall, and f1-score which is harmonics mean of

© ISO/IEC 2026 – All rights reserved
precision and recall. Also, sample numbers that are used for calculation are also important indicator to
discuss effectiveness of the metrics.
— Validity of trajectory, which is an indicator for evaluating degree of incursion of the trajectories into un-
walkable area defined in floor map. Validity of trajectory can be evaluated without GT. This metric shall
be regarded as a negative check.
6.3 Indicators related to series of positional estimation
This subclause presents, indicators related to series of positional estimation, which is type of indicator
calculated by using series of estimation of position. If the benchmarking organizer or datasets include GT
with continuous and frequent positional estimation, this type of errors can be evaluated.
— Velocity error (VE), which is the error of velocity compared with correct velocity. Velocity can be
calculated by time derivative of position. Velocity is not a concern only for xDR. However, velocity is
important especially for xDR. This metric is regarded as a PIEM, because this is independent to the path.
— Distance error (DE), which is the error of the distance travelled of a tracking target. The distance
travelled is a useful indicator for estimating physical demand of workers or wear of moving vehicle.
Assuming the simplest case in which the estimated distance is accumulation of location and GT of the
distance travelled is available, the distance error can be measured as difference between the estimated
distance and the true distance. The error can be normalized by the true distance.
— Moving velocity check, which is a checking items if estimated local moving speed is in the decent range
assumed for the target. For example, 1,5 m/s is an example threshold of walking speed of human. VE
is also evaluation indicators for velocity by comparing with GT of speed. Moving velocity check can be
evaluated without GT. Moving velocity check shall be regarded as a negative check.
6.4 Indicators related to orientation
This subclause presents indicators related to orientation, which is type of indicator related to the error of
orientation estimation by LTMs. Some of LTMs, such as methods integrating with xDR, have a capability for
estimating heading direction. If the benchmarking organizer or datasets have GT of heading direction, this
type of errors can be evaluated.
— Orientation error accumulation gradient (OEAG) is speed for orientation error accumulation caused by
relative tracking, an example of the EAG and PIEM. This evaluation metric can be calculated if correct
orientation of the target is available. There are multiple derivatives of OEAG according to the difference
of normalization.
— Time normalized orientation error accumulation gradient (T-OEAG).
— Distance normalized orientation error accumulation gradient (D-OEAG).
— Angular change normalized orientation error accumulation gradient (AC-OEAG).
6.5 Indicators related to relative relationship
This subclause presents indicators related to relative relationship, which is type of the indicator related
to error of positional and orientational relative relationship of two tracking targets. These indicators can
contain several indicators to evaluate different aspects of relative relationship. These indicators can be
normalized by the relative distance for weighting nearer tracking targets.
— Relative distance accuracy, or "relative accuracy" as introduced in ISO/IEC 18305, measures error of the
estimated distance between two tracking targets compared with correct distance between them.
— Relative pose accuracy measures the error of angle difference of angles relative to the line passing
through the two tracking targets between tracking targets. This indicator is useful for estimating error
in measuring interaction of the two tracking targets, such as face-to-face communication.
Figure 6 shows examples of the calculation of the indicators related to relative relationship.

© ISO/IEC 2026 – All rights reserved
Figure 6 — Examples of calculation for indicators related to relative relationship
6.6 Indicator related to integrated localization
This subclause presents an indicator related to integrated localization, which is a type of indicator estimated
for measuring performance in the integration of the estimate by xDR with other localization methods.
— ALIP and ALAP based indicators, which is a type of the indicators utilized operation on data in ALIP.
Dependency of AL can be evaluated by comparing performance in ALIP and ALAP.
6.7 Indicators related to practicability of localization system
This subclause presents supplemental indicators for practicability of localization system, which is type of
indicator not directly related to localization performance but related to practicality of localization system.
Set-up time, costs, set-up easiness, availability are introduced with similar consideration in ISO/IEC 18305.
— Availability, which is the ratio of expected number of outputs from LTMs and actual available number
of LTM output. For example, a system that only outputs confident location results and discards
uncertain output, which leads to reduced output number. Interval of output is used as an evaluation
criterion whether the system continuously outputs or not. Threshold for interval of output is defined
by considering minimal requirements of the system. Availability shall be regarded as negative checks
because continuous output is a requirement for practical localization methods.
ISO/IEC 18520 introduced through-put and latency as the temporality indicators. They are necessary to
discuss real-time issue. Same concept shall be applicable for benchmarking of LTMs.
6.8 Indicators related to data used for the benchmarking
This subclause presents indicators related to data used for the benchmarking. ISO/IEC 18520 introduced
number of datasets and variety on properties of datasets as the variety indicators. They are introduced
for preventing fine tuning and cheating with some specific dataset. Same concept shall be applicable for
benchmarking of LTMs.
7 Master set for benchmarking
7.1 Overview
This clause identifies the reference elements in a master set used for benchmarking. Table 3 shows
representative elements in a master set for off-site benchmarking.

© ISO/IEC 2026 – All rights reserved
Table 3 — Dataset for benchmarking
Category Contents
Contents of dataset for Inertial measurement unit (IMU) data, Bluetooth low energy (BLE)/
indoor localization WiFi signals received by known access points (APs) or beacons, floor
map, BLE beacon/WiFi APs’ location, absolute localization inapplicable
period (ALIP)/ Absolute localization applicable period (ALAP), correc-
tion points, trial data, scoring data
Ground-truth (GT) used GT for trial-data (open), GT for evaluation (closed), how to collect the
for evaluation ground truth data, contents of GT, frequency of GT, accuracy of GT
Supplemental information Scenario, motion type, environment (size), # of target, length of the
for clarifying the situa- dataset, mearing device specification, how to mount the device, how to
tion of the dataset sync the devices, calibration data for IMU calibration
7.2 Dataset for off-site benchmarking
This subclause presents representative elements such as contents and supplemental information in datasets
for off-site benchmarking.
— Contents of dataset for indoor localization, which include:
— Sensor data required for integrated indoor localization using xDR: inertial measurement unit (IMU)
data (acceleration, angular velocity, magnetism), atmospheric pressure, either BLE signal or Wi-Fi
signals, or both.
— Supplemental information required for integrated indoor localization: floor map, location of BLE
tags and Wi-Fi access points, information about ALIP and ALAP.
— GT, which include:
— GT for testing trial data (provided for the participant of benchmarking).
— GT for scoring trial data (not provided for the participants, used for the evaluation).
— Specification of the GT (method for data collection, sampling frequency of GT, accuracy of GT).
— Supplemental information for clarifying the situation of the dataset, which include:
— Scenario of dataset.
— Tracking target:
— Motion type of the target, number of targets, height and weight of the target (if the target is
pedestrian).
— Length of the dataset:
— Measuring device (specification of devices, how to mount the divide, synchronization method,
calibration for the devices).
— Environment in where the dataset is measured (type, size, etc.).
8 Conformance
Table 4 shows an example of a conformance checklist. This checklist or customized ones shall be used to
clarify how each benchmarking activity conforms to this document in a compact form. This checklist is
useful to summarize and declare which datasets, benchmark indicators are included, and what types of trial
sets are used in each benchmarking activity.

© ISO/IEC 2026 – All rights reserved
Table 4 — Conformance checklist
Category Sub-category Items Check / Value
Dataset Specification of the dataset Scenario of the dataset
Movement contained, size of the target area.
Scale of the dataset (length of the dataset)
Format and contents of the dataset
Device used for the measuring dataset
Ground-truth How to collect ground-truth
Accuracy and precision of the ground truth
(reference data)
Frequency of the ground-truth
Contents of the ground-truth data
Evaluation Indicators related to position PEAG (time/distance/angular change normal-
metrics ized)
Circular error
Circular accuracy (in world/local coordinate
systems)
Area detection performance
Validity of trajectory
Indicators related to series of Velocity error (VE)
positional estimation
Distance error
Moving velocity check
Indicators related to orientation OEAG (time/distance/angular change normal-
ized OEAG)
Indicators related to relative Relative distance accuracy
relationship
Relative pose accuracy
Indicator related to integrated Difference between the performances in ALIP
localization and ALAP
Indicators related to practicabil- Set-up time, cost, easiness for users, availability,
ity of localization system through-put, Latency
Indicators related to data used Number of datasets, variety on properties of
for the benchmarking datasets
© ISO/IEC 2026 – All rights reserved
Annex A
(informative)
Benchmarking activities
A.1 xDR Challenge 2023
A.1.1 Overview of xDR Challenge 2023
xDR Challenge is a series of indoor localization competitions by the PDR benchmark standardization,
mainly focused on integrated indoor localization methods with dead reckoning. xDR Challenge 2023 was
[6]
an official competition track of IPIN competition. The competition organizer provided datasets for indoor
localization to competitors, and the competitors are asked to submit estimated trajectories of targets in
certain time limit.
Similar to the previous PDR and xDR Challenges, submitted trajectories were evaluated by multi-faced
evaluation metrics. The xDR Challenge 2023 was conducted closely together with other tracks of the IPIN
competitions under a common schedule. The xDR Challenge adopted common tool named EvAAL API
(application programming interface) for sharing dataset and receiving results. As aligned with other tracks,
scoring trial was conducted at a day in the middle of September 2023. The competition organizer provided
training data and testing trials in same format with the real dataset for allowing the competitors to prepare
and adjust localization algorithm and systems before the real competition.
A conformance check list for the xDR Challenge 2023 is attached as Annex B.1.
A.1.2 Dataset for xDR Challenge 2023
THE Target field of xDR Challenge 2023 was commercial facilities and the targets of localization were
pedestrian walking in the field. tHE Competition organizer arranged totally about 100 BLE beacons in the
field for correcting error of PDR. A light detection and ranging (LiDAR) device was adopted for collecting
GT of position of the targets. Competition organizer evaluated the estimated trajectories of the target by
utilizing multi-faceted evaluation metrics with the precise GT data by LiDAR.
The datasets of the xDR Challenge 2023 consisted of sensor data required for the PDR-based indoor
localization as shown in Figure A.1. The data were measured in the commercial facilities in a highway rest
spot. Competition organizer measured pedestrian movement by using Android devices. The competition
organizers collected sensor (gyro, accelerometer, magnets sensor) data as well as the received signal
strength indicators (RSSIs) from the BLE beacons. Table A.1 shows the specification contents of the dataset
for xDR Challenge 2023. Figure A.2 shows the method for synchronization for the dataset and GT for this
dataset. Network time protocol (NTP) server was used for synchronizing smartphones’ sensor data. An
Android smartphone (AQUOS #5) and a handheld LiDAR were synchronized by finding the best time offset
which had the maximum correlation of angular velocity.

© ISO/IEC 2026 – All rights reserved
Figure A.1 — Dataset of xDR Challenge
Figure A.2 — Sensor synchronization methods for xDR Challenge 2023

© ISO/IEC 2026 – All rights reserved
Table A.1 — Specification of contents of the dataset for xDR Challenge 2023
Available in Available in
Data type Measuring device Rate
training data scoring trials
Acceleration AQUOS Sense 6 Approx. 100 Hz Yes Yes
Angular velocity AQUOS Sense 6 Approx. 100 Hz Yes Yes
Magnetism AQUOS Sense 6 Approx. 100 Hz Yes Yes
BLE RSSI AQUOS Sense 6 Emitted from beacons Yes Yes
at 10 Hz, recorded
when received by
AQUOS Sense 6.
Ground truth loca- ZEB-Horizon Approx. 100 Hz Yes Only start and
tion (x, y, z) end
Ground truth ori- ZEB-Horizon Approx. 100 Hz Yes Only start and
entation (quater- end
nion)
Ground truth floor - 1 floor name for each Yes Yes
name path
A.1.3 Evaluation for xDR Challenge 2023
xDR Challenge 2023 was an indoor localization competition held as an official competition track of
international conference on indoor position and indoor navigation (IPIN) 2023. The evaluation framework
described in this document were applied. In the xDR Challenge 2023, submitted trajectories were evaluated
by multi-faced evaluation metri
...