ISO/IEC 15938-18:2023

INTERNATIONAL ISO/IEC
STANDARD 15938-18
First edition
2023-07
Information technology — Multimedia
content description interface —
Part 18:
Conformance and reference software
for compression of neural networks
Technologies de l'information — Interface de description du contenu
multimédia —
Partie 18: Conformité et logiciel de référence pour la compression des
réseaux neuronaux
Reference number
© ISO/IEC 2023
© ISO/IEC 2023
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on
the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below
or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
ii
© ISO/IEC 2023 – All rights reserved

Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
4 Conformance testing . .1
4.1 General . 1
4.2 Conformance testing for decoder . 2
4.3 Conformance testing for bitstreams . 2
4.4 Models and reference bitstreams . 2
4.5 Procedure to test decoders . 7
4.5.1 General . 7
4.5.2 Decoding self-contained NNC bitstreams . 8
4.5.3 Decoding NNC bitstreams using out-of-band parameters . 8
4.6 Procedure to test bitstreams . 8
5 Reference software . 8
5.1 General . 8
5.2 Software location and license . 9
5.3 Software installation . 9
5.4 Software architecture . 9
5.4.1 General . 9
5.4.2 Parameter reduction methods . 10
5.4.3 Parameter approximation . 10
5.4.4 Reconstruction . . . 10
5.4.5 Encode. 10
5.4.6 Decode . 11
5.5 Data structures and interfaces . 11
5.5.1 model_info: Shared model information . 11
5.5.2 approx_data – Data structure for interface #4 .12
5.5.3 nctm – Main module . 14
5.5.4 nctm.nnr_model – Module for handling model related functionalities . 17
Annex A (informative) Implementation in Python.21
Bibliography .22
iii
© ISO/IEC 2023 – All rights reserved

Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical
activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international
organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the
work.
The procedures used to develop this document and those intended for its further maintenance
are described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria
needed for the different types of document should be noted. This document was drafted in
accordance with the editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives or
www.iec.ch/members_experts/refdocs).
Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent
rights. Details of any patent rights identified during the development of the document will be in the
Introduction and/or on the ISO list of patent declarations received (see www.iso.org/patents) or the IEC
list of patent declarations received (see https://patents.iec.ch).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and
expressions related to conformity assessment, as well as information about ISO's adherence to
the World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see
www.iso.org/iso/foreword.html. In the IEC, see www.iec.ch/understanding-standards.
This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 29, Coding of audio, picture, multimedia and hypermedia information.
A list of all parts in the ISO/IEC 15938 series can be found on the ISO and IEC websites.
Any feedback or questions on this document should be directed to the user’s national standards
body. A complete listing of these bodies can be found at www.iso.org/members.html and
www.iec.ch/national-committees.
iv
© ISO/IEC 2023 – All rights reserved

Introduction
This document describes conformance testing and the reference software for ISO/IEC 15938-17
Compression of neural networks for multimedia content description and analysis. The reference
software includes both encoder and decoder functionality.
The reference software is useful in aiding users of a standard for coding neural networks to establish
and test conformance and interoperability, and to educate users and demonstrate the capabilities of
the standard. For these purposes, the accompanying software is provided as an aid for the study and
implementation of 15938-17 compression of neural networks for multimedia content description and
analysis.
The purpose of this document is to provide the following:
— A set of reference bitstreams conforming to ISO/IEC 15938-17.
— Description of procedures to test conformance of bitstreams and decoders to ISO/IEC 15938-17.
— Reference decoder software capable of decoding bitstreams that conform to ISO/IEC 15938-17 in a
manner that conforms to the decoding process specified in ISO/IEC 15938-17.
— Reference encoder software capable of producing bitstreams that conform to ISO/IEC 15938-17.
v
© ISO/IEC 2023 – All rights reserved

INTERNATIONAL STANDARD ISO/IEC 15938-18:2023(E)
Information technology — Multimedia content description
interface —
Part 18:
Conformance and reference software for compression of
neural networks
1 Scope
This document specifies conformance testing procedures for implementations of ISO/IEC 15938-17 and
provides conformance bitstreams. It also provides the reference software for ISO/IEC 15938-17 which
is an integral part of this document.
2 Normative references
The following documents are referred to in the text in such a way that some or all of their content
constitutes requirements of this document. For dated references, only the edition cited applies. For
undated references, the latest edition of the referenced document (including any amendments) applies.
ISO/IEC 15938-17:2022, Information technology — Multimedia content description interface — Part 17:
Compression of neural networks for multimedia content description and analysis
ISO/IEC 21778, Information technology — The JSON data interchange syntax
3 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO/IEC 15938-17 and the
following apply.
ISO and IEC maintain terminology databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at https:// www .electropedia .org/
3.1
array
ordered list of elements where all elements are of the same type
3.2
dictionary
ordered list of key/value pairs where each key/value pair is an list of two elements with the first element
being denoted ‘key’ and with the second element being denoted ‘value’
4 Conformance testing
4.1 General
Subclauses 4.2 through 4.6 specify tests for verifying the conformance of bitstreams as well as decoders.
These tests make use of test data (bitstreams and related models) provided at https:// standards .iso
© ISO/IEC 2023 – All rights reserved

.org/ iso -iec/ 15938/ -18/ ed -1/ en (described in detail in 4.4) and follow the procedure described in
subclause 4.5.
4.2 Conformance testing for decoder
The decoder conformance is specified in ISO/IEC 15938-17:2022, Clause 7.
4.3 Conformance testing for bitstreams
The bitstream conformance is specified in ISO/IEC 15938-17:2022, Clause 7.
4.4 Models and reference bitstreams
A set of bitstreams and related neural network models is provided for conformance testing. When neural
network models are provided, they are the source models used to generate one or more compressed
bitstreams. The code defining these models is provided as part of the reference software distribution at
https:// standards .iso .org/ iso -iec/ 15938/ -18/ ed -1/ en. A bitstream can be generated:
— from a commonly used neural network model,
— from synthetic data by encoding it with the reference encoder,
— by creating a conformant synthetic bitstream directly, without using an encoder.
Where applicable, the dataset used to train the model is provided for information only. However, the
dataset is not needed for conformance testing and thus not provided.
Table 1 summarizes the provided bitstreams. It also lists the features tested with the bitstreams, and
the reference encoder configuration used to generate the bitstream (where applicable). If decoding the
bitstream requires out-of-band parameters (e.g. information that
...