ISO/IEC 18520:2019

INTERNATIONAL ISO/IEC
STANDARD 18520
First edition
2019-01
Information technology — Computer
graphics, image processing and
environmental data representation —
Benchmarking of vision-based spatial
registration and tracking methods for
mixed and augmented reality (MAR)
Technologies de l'information — Infographie, traitement d'images
et représentation des données environnementales — Étalonnage des
méthodes d'enregistrement géométriques et de suivi basées sur la
vision pour le MAR
Reference number
©
ISO/IEC 2019
© ISO/IEC 2019
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting
on the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address
below or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Fax: +41 22 749 09 47
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
ii © ISO/IEC 2019 – All rights reserved

Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Normative references . 1
3 Terms, definitions, acronyms and abbreviated terms . 1
3.1 Terms and definitions . 1
3.2 Acronyms and abbreviated terms . 3
4 Overview of the framework . 3
5 Benchmarking processes . 4
5.1 Overview . 4
5.2 Process and process flow . 5
5.3 Stakeholders . 6
6 Benchmark indicators . 7
6.1 Overview . 7
6.2 Reliability indicators . 8
6.3 Temporality indicators . 8
6.4 Variety indicators. 9
7 Trial set for benchmarking .10
7.1 Overview .10
7.2 Dataset for on- and off-site benchmarking .10
7.3 Physical object instances for on- and off-site benchmarking .11
8 Conformance .12
Annex A (informative) Benchmarking activities .14
Annex B (informative) Usage examples of conformance checklists .39
Annex C (informative) Conceptual relationship between this document and other standards .59
Bibliography .61
© ISO/IEC 2019 – All rights reserved iii

Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that
are members of ISO or IEC participate in the development of International Standards through
technical committees established by the respective organization to deal with particular fields of
technical activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other
international organizations, governmental and non-governmental, in liaison with ISO and IEC, also
take part in the work.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for
the different types of document should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www .iso .org/directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent
rights. Details of any patent rights identified during the development of the document will be in the
Introduction and/or on the ISO list of patent declarations received (see www .iso .org/patents) or the IEC
list of patent declarations received (see http: //patents .iec .ch).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation on the voluntary nature of standards, the meaning of ISO specific terms and
expressions related to conformity assessment, as well as information about ISO's adherence to the
World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see www .iso
.org/iso/foreword .html.
This document was prepared by Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 24, Computer graphics, image processing and environmental data representation.
Any feedback or questions on this document should be directed to the user’s national standards body. A
complete listing of these bodies can be found at www .iso .org/members .html.
iv © ISO/IEC 2019 – All rights reserved

Introduction
In the development of mixed and augmented reality (MAR) applications, one of the most important
technologies involves spatial registration and spatial tracking methods, especially vision-based
methods. The research and development on registration and tracking based on computer vision
technologies is flourishing and many new algorithms have been proposed every year.
Therefore, this document aims at fostering objective evaluation and comparison of diverse registration
and tracking methods, in order to facilitate fairer competition among small and major companies/
institutes involved in MAR technologies, applications and services.
Moreover, this document can be the baseline to standardize spatial registration and tracking methods
which not only utilize a video camera but combine a video camera with other sensors such as another
video camera, a depth camera, inertial sensors and infrastructure-based positioning technologies,
and which utilize technologies such as IoT (Internet of Things) and GNSS (Global Navigation Satellite
System).
The target audience of this document includes stakeholders of benchmarking activities. The following
are examples of how this document can be used directly or indirectly:
— by a benchmarking service provider, a benchmark provider or a benchmarking competition organizer
who wishes to align their benchmarking activities including self-benchmarking and open/closed
competitions to be consistent with this document;
— by a technology developer/supplier who wishes to estimate and evaluate the performance of a vision-
based spatial registration and tracking (vSRT) method for MAR appropriately with a benchmarking
service provider, a benchmark provider or a benchmarking competition organizer who aligns their
benchmarking activities to be consistent with this document; or
— by a technology user who wishes to obtain benchmarking results based on a benchmarking activity,
which is consistent with this document, or to compare the existing vSRT methods for MAR in terms
of their performance.
© ISO/IEC 2019 – All rights reserved v

INTERNATIONAL STANDARD ISO/IEC 18520:2019(E)
Information technology — Computer graphics, image
processing and environmental data representation —
Benchmarking of vision-based spatial registration and
tracking methods for mixed and augmented reality (MAR)
1 Scope
This document identifies the reference framework for the benchmarking of vision-based spatial
registration and tracking (vSRT) methods for mixed and augmented reality (MAR).
The framework provides typical benchmarking processes, benchmark indicators and trial set elements
that are necessary to successfully identify, define, design, select and apply benchmarking of vSRT
methods for MAR. It also provides definitions for terms on benchmarking of vSRT methods for MAR.
In addition, this document provides a conformance checklist as a tool to clarify how each benchmarking
activity conforms to this document in a compact form by declaring which benchmarking processes and
benchmark indicators are included and what types of trial sets are used in each benchmarking activity.
2 Normative references
There are no normative references in this document.
3 Terms, definitions, acronyms and abbreviated terms
For the purposes of this document, the following terms and definitions apply.
ISO and IEC maintain terminological databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https: //www .iso .org/obp
— IEC Electropedia: available at http: //www .electropedia .org/
3.1 Terms and definitions
3.1.1
benchmark
reference point against which comparisons can be made
Note 1 to entry: In the context of this document, the performance of vSRT methods for MAR is the object of
comparison.
Note 2 to entry: See ISO/IEC 29155-1.
3.1.2
benchmark indicator
indicator that qualitatively shows a particular aspect of a benchmark (3.1.1) with appropriate metrics
3.1.3
benchmarking
activity of comparing objects of interest to each other or against benchmarks (3.1.1) to evaluate relevant
characteristics
Note 1 to entry: In the context of this document, the object of interest is the performance of vSRT methods for MAR,
and the characteristics are particular aspects of the performance such as reliability, temporal characteristic, etc.
© ISO/IEC 2019 – All rights reserved 1

Note 2 to entry: See ISO/IEC 29155-1.
3.1.4
benchmarking instrument
tool, method or guide used to support every activity within the benchmarking (3.1.3) framework
3.1.5
benchmarking method
particular procedure for conducting benchmarking (3.1.3) and obtaining benchmarking results (3.1.7)
3.1.6
benchmarking repository
repository which is designated for retaining information necessary for benchmarking (3.1.3) such as
datasets (3.1.9) for benchmarking and benchmarking results (3.1.7)
Note 1 to entry: Some benchmarking repositories might contain all information, and some might contain a subset.
Note 2 to entry: See ISO/IEC 29155-1.
3.1.7
benchmarking results
benchmarks (3.1.1) as primary results, and intermediate results and reports on benchmarking (3.1.3)
including findings, issues and lessons learned as secondary results
Note 1 to entry: In the context of this document, a typical intermediate result is the output of vSRT methods such
as the estimated position of a camera the accuracy of which is evaluated with a benchmark indicator.
3.1.8
competition organizer
person or organization that hosts an on-site (3.1.14) or off-site (3.1.13) benchmarking competition
3.1.9
dataset
collection of data that contains target images and the ground truth (3.1.11) regarding the target images
for benchmarking (3.1.3)
3.1.10
extrinsic camera parameters
parameters that define the positio
...