ISO/IEC TR 19566-1:2016

TECHNICAL ISO/IEC TR
REPORT 19566-1
First edition
Information technology — JPEG
Systems —
Part 1:
Packaging of information using
codestreams and file formats
Technologies de l’information — Systèmes JPEG —
Partie 1: Emballage des informations utilisant les flux de codes et les
formats de fichier
PROOF/ÉPREUVE
Reference number
ISO/IEC TR 19566-1:2016(E)
©
ISO/IEC 2016

---------------------- Page: 1 ----------------------
ISO/IEC TR 19566-1:2016(E)

COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2016, Published in Switzerland
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized otherwise in any form
or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior
written permission. Permission can be requested from either ISO at the address below or ISO’s member body in the country of
the requester.
ISO copyright office
Ch. de Blandonnet 8 • CP 401
CH-1214 Vernier, Geneva, Switzerland
Tel. +41 22 749 01 11
Fax +41 22 749 09 47
copyright@iso.org
www.iso.org
ii © ISO/IEC 2016 – All rights reserved

---------------------- Page: 2 ----------------------
ISO/IEC TR 19566-1:2016(E)

Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Terms, definitions, abbreviated terms and symbols . 1
2.1 Terms and definitions . 1
2.2 Symbols . 5
2.3 Abbreviated terms . 5
3 Conventions . 6
3.1 Operators . 6
3.1.1 Arithmetic operators . 6
3.1.2 Logical operators . 7
3.1.3 Relational operators . 7
3.1.4 Precedence order of operators . 7
3.1.5 Mathematical functions . 8
4 General . 8
5 Background . 8
5.1 General . 8
5.2 Organization of information .10
Annex A (informative) Organization of information — Signalling, packaging and storage .11
Annex B (informative) Concept of boxes .13
Annex C (informative) Box-based file format structure .16
Annex D (informative) Codestream syntax .19
Annex E (informative) Codec/codestream functionality guidelines .22
Bibliography .26
© ISO/IEC 2016 – All rights reserved PROOF/ÉPREUVE iii

---------------------- Page: 3 ----------------------
ISO/IEC TR 19566-1:2016(E)

Foreword
ISO (the International Organization for Standardization) is a worldwide federation of national standards
bodies (ISO member bodies). The work of preparing International Standards is normally carried out
through ISO technical committees. Each member body interested in a subject for which a technical
committee has been established has the right to be represented on that committee. International
organizations, governmental and non-governmental, in liaison with ISO, also take part in the work.
ISO collaborates closely with the International Electrotechnical Commission (IEC) on all matters of
electrotechnical standardization.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for the
different types of ISO documents should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject of
patent rights. ISO shall not be held responsible for identifying any or all such patent rights. Details of
any patent rights identified during the development of the document will be in the Introduction and/or
on the ISO list of patent declarations received (see www.iso.org/patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation on the meaning of ISO specific terms and expressions related to conformity
assessment, as well as information about ISO’s adherence to the WTO principles in the Technical
Barriers to Trade (TBT) see the following URL: Foreword - Supplementary information
The committee responsible for this document is ISO/IEC JTC 1, Information technology, Subcommittee
SC 29, Coding of audio, picture, multimedia and hypermedia information.
iv PROOF/ÉPREUVE © ISO/IEC 2016 – All rights reserved

---------------------- Page: 4 ----------------------
ISO/IEC TR 19566-1:2016(E)

Introduction
This part of ISO/TR 19566 provides an overview for users of ISO/IEC standards in the use of
common JPEG syntax elements at the systems layer for current and future International Standards
developed by ISO/IEC.
With the development of coding technologies, ISO/IEC has defined a number of different file formats
and multiple variants of codestream syntax. Many of these are specialized to dedicated use cases
or compression algorithms. Consequently, it is difficult to maintain an overview about existing file
formats, their capabilities and their architectures.
This part of ISO/TR 19566 aims to describe common architectural concepts for file formats and
codestream formats. By these means, it lays out guidelines for future file formats and codestream
syntax. By observing these guidelines, future International Standards may fit into an overall operable
Systems infrastructure that can handle all tools standardized by the ISO/IEC body.
© ISO/IEC 2016 – All rights reserved PROOF/ÉPREUVE v

---------------------- Page: 5 ----------------------
TECHNICAL REPORT ISO/IEC TR 19566-1:2016(E)
Information technology — JPEG Systems —
Part 1:
Packaging of information using codestreams and file
formats
1 Scope
This part of ISO/TR 19566 describes common elements of a system layer for JPEG standards, referred to
as JPEG Systems.
This part of ISO/TR 19566 describes the common architecture of file formats and codestream formats
used in JPEG standards. It is intended that all future Systems components support codestreams and file
formats following these guidelines.
2 Terms, definitions, abbreviated terms and symbols
2.1 Terms and definitions
For the purposes of this document, the following terms and definitions apply.
2.1.1
backward compatibility
inclusive of the old specification within the new specification
Note 1 to entry: Any devices implementing the new standard can also interpret all data compliant with the old
version of the standard. However, an old device only compliant with the old version of the standard might not be
able to interpret the data compliant with the new version of the standard.
2.1.2
bit stream
partially encoded or decoded sequence of bits comprising an entropy-coded segment
2.1.3
box
structured collection of data describing the image or the image decoding process
Note 1 to entry: See B.2 for the definition of boxes.
2.1.4
box-based file format
file format whose composing elements are well-defined, hierarchically structured boxes
2.1.5
byte
group of 8 bits
2.1.6
coder
embodiment of a coding process
© ISO/IEC 2016 – All rights reserved PROOF/ÉPREUVE 1

---------------------- Page: 6 ----------------------
ISO/IEC TR 19566-1:2016(E)

2.1.7
coding
umbrella term that defines both the encoding/compression of a signal as well as the
decoding/decompression of a signal
2.1.8
coding model
procedure used to convert input data into symbols to be coded
2.1.9
coding process
process which transforms compressed data into a continuous-tone image and/or a continuous-tone
image into its compressed representation
Note 1 to entry: It presents thus an umbrella term for “encoding process” and “decoding process”.
2.1.10
compression
reduction in the number of bits used to represent source image data
2.1.11
component
two-dimensional array of samples having the same designation in the output or display device
Note 1 to entry: An image typically consists of several components, for example, red, green and blue.
2.1.12
continuous-tone image
image whose components have more than one bit per sample
2.1.13
decoder
embodiment of a decoding process
3.1.14
decoding process
process which takes as its input compressed image data and outputs a continuous-tone image
2.1.15
dequantization
inverse procedure to quantization by which the decoder recovers a representation of the DCT coefficients
2.1.16
downsampling
procedure by which the spatial resolution of a component is reduced
2.1.17
encoder
embodiment of an encoding process
2.1.18
encoding process
process which takes as its input a continuous-tone image and outputs compressed image data
2.1.19
entropy-coded (data) segment
independently decodable sequence of entropy encoded bytes of compressed image data
2.1.20
entropy decoder
embodiment of an entropy decoding procedure
2 PROOF/ÉPREUVE © ISO/IEC 2016 – All rights reserved

---------------------- Page: 7 ----------------------
ISO/IEC TR 19566-1:2016(E)

2.1.21
entropy decoding
lossless procedure which recovers the sequence of symbols from the sequence of bits produced by the
entropy encoder
2.1.22
entropy encoder
embodiment of an entropy encoding procedure
2.1.23
entropy encoding
lossless procedure which converts a sequence of input symbols into a sequence of bits such that the
average number of bits per symbol approaches the entropy of the input symbols
2.1.24
forward compatibility
only compliant with the old specification but able to interpret the new specification
Note 1 to entry: Although devices only compliant with the old version of the standard are nevertheless able to
interpret the data conforming with the new standard, it is possible that the obtained results are not as good as
when using a device compliant with the new version of the standard.
2.1.25
grayscale image
continuous-tone image that has only one component
2.1.26
high dynamic range
image or image data comprised of more than eight bits per sample, coded in floating point representation
2.1.27
intermediate dynamic range
image or image data comprised of more than eight bits per sample
2.1.28
Joint Photographic Experts Group
JPEG
informal name of the committee which created this part of ISO/TR 19566
2.1.29
JPEG standards
collection of ISO/IEC/ITU standards developed by the Joint Photographic Experts Group for still imaging
application as listed in the Bibliography
2.1.30
JPEG Systems
common elements of a system layer for JPEG standards
2.1.31
lossless
descriptive term for encoding and decoding processes and procedures in which the output of the
decoding procedure(s) is identical to the input to the encoding procedure(s)
2.1.32
lossless coding
mode of operation which refers to any one of the coding processes defined in this part of ISO/TR 19566
in which all of the procedures are lossless
2.1.33
lossy
descriptive term for encoding and decoding processes which are not lossless
© ISO/IEC 2016 – All rights reserved PROOF/ÉPREUVE 3

---------------------- Page: 8 ----------------------
ISO/IEC TR 19566-1:2016(E)

2.1.34
low-dynamic range
image or image data comprised of data with no more than eight bits per sample
2.1.35
marker
two-byte code in which the first byte is hexadecimal FF and the second byte is a value between 1 and
hexadecimal FE
2.1.36
marker segment
marker together with its associated set of parameters
2.1.37
metadata
additional data associated with the image data beyond the image data
2.1.38
minimum coded unit
MCU
smallest group of data units that is coded
2.1.39
pixel
collection of sample values in the spatial image domain having all the same sample coordinates
EXAMPLE A pixel may consist of three samples describing its red, green and blue value.
2.1.40
point transform
scaling of a sample or DCT coefficient by a factor
2.1.41
precision
number of bits allocated to a particular sample or DCT coefficient
2.1.42
procedure
set of steps which accomplishes one of the tasks which comprise an encoding or decoding process
2.1.43
quantization value
integer value used in the quantization procedure
2.1.44
quantize
act of performing the quantization procedure for a DCT coefficient
2.1.45
sample
one element in the two-dimensional image array which comprises a component
2.1.46
sample grid
common coordinate system for all samples of an image
Note 1 to entry: The samples at the top left edge of the image have the coordinates (0,0), the first coordinate
increases towards the right, the second towards the bottom.
4 PROOF/ÉPREUVE © ISO/IEC 2016 – All rights reserved

---------------------- Page: 9 ----------------------
ISO/IEC TR 19566-1:2016(E)

2.1.47
table specification data
coded representation from which the tables used in the encoder and decoder are generated and their
destinations specified
2.1.48
(uniform) quantization
procedure by which DCT coefficients are linearly scaled in order to achieve compression
2.1.49
upsampling
procedure by which the spatial resolution of a component is increased
2.1.50
vertical sampling factor
relative number of vertical data units of a particular component with respect to the number of vertical
data units in the other components in the frame
2.1.51
zero byte
0x00 byte
2.2 Symbols
X Width of the sample grid in positions
Y Height of the sample grid in positions
Nf Number of components in an image
s Subsampling factor of component i in horizontal direction
i,x
s Subsampling factor of component i in vertical direction
i,y
2.3 Abbreviated terms
For the purposes of this document, the following abbreviated terms apply.
API Application Programming Interface
AR Augmented Reality
ASCII American Standard Code for Information Interchange
DCT Discrete Cosine Transformation
EXIF Exchangeable Image File Format
HDR High Dynamic Range
IDR Intermediate Dynamic Range
JBIG Joint Bi-level Image experts Group
JFIF JPEG File Interchange Format
JP2 JPEG 2000 file format
JPEG Joint Photographic Experts Group
© ISO/IEC 2016 – All rights reserved PROOF/ÉPREUVE 5

---------------------- Page: 10 ----------------------
ISO/IEC TR 19566-1:2016(E)

JPSec JPEG 2000 Secured
JPIP JPEG 2000 Interactive Protocol
JPWL JPEG 2000 Wireless
JPX Extended JPEG 2000 file format
LDR Low Dynamic Range
LSB Least Significant Bit
MJ2 Motion JPEG 2000 file format
MJPEG2000 Motion JPEG 2000
MSB Most Significant Bit
SPIFF Still Picture Interchange File Format
TIFF Tagged Image File Format
TMO Tone Mapping Operator
XML Extensible Markup Language
3 Conventions
3.1 Operators
NOTE Many of the operators used in this part of ISO/TR 19566 are similar to those used in the C
programming language.
3.1.1 Arithmetic operators
+ Addition
− Subtraction (as a binary operator) or negation (as a unary prefix operator)
* Multiplication
/ Division without truncation or rounding
smod
   
x smod a is the unique value y between −−a1 /2 and a −12/
() ()
   
   
for which y + Na = x with a suitable integer N.
umod x mod a is the unique value y between 0 and a−1
for which y + Na = y with a suitable integer N.
6 PROOF/ÉPREUVE © ISO/IEC 2016 – All rights reserved

---------------------- Page: 11 ----------------------
ISO/IEC TR 19566-1:2016(E)

3.1.2 Logical operators
|| Logical OR
&& Logical AND
! Logical NOT
∈ x ∈ {A, B} is defined as (x == A || x == B)
∉ x ∉ {A, B} is defined as (x != A && x != B)
3.1.3 Relational operators
> Greater than
>= Greater than or equal to
< Less than
<= Less than or equal to
== Equal to
!= Not equal to
3.1.4 Precedence order of operators
Operators are listed below in descending order of precedence. If several operators appear in the same
line, they have equal precedence. When several operators of equal precedence appear at the same level
in an expression, evaluation proceeds according to the associativity of the operator either from right to
left or from left to right.
Operators Type of operation Associativity
(), [], . Expression Left to right
− Unary negation
*, / Multiplication Left to right
umod, smod Modulo (remainder) Left to right
+, − Addition and subtraction Left to right
<, >, <=, >= Relational Left to right
© ISO/IEC 2016 – All rights reserved PROOF/ÉPREUVE 7

---------------------- Page: 12 ----------------------
ISO/IEC TR 19566-1:2016(E)

3.1.5 Mathematical functions
Ceil of x; returns the smallest integer that is greater than or equal to x.
 
x
 
Floor of x; returns the largest integer that is lesser than or equal to x.
 
x
 
|x| Absolute value, is –x for x < 0, otherwise x.
sign(x) Sign of x, zero if x is zero, +1 if x is positive, −1 if x is negative.
clamp(x,min,max) Clamps x to the range [min,max]: returns min if x < min, max if x > max or other-
wise x.
power(x,a) Raises the value of x to the power of a. x is a non-negative real number, a is a real
number. Power(x,a) is equal to exp(a×log(x)) where exp is the exponential func-
tion and log() is the natural logarithm. If x is zero and a is positive, power(x,a) is
defined to be zero.
4 General
The purpose of this Clause is to give an informative overview of the elements specified in this part of
ISO/TR 19566. Another purpose is to introduce many of the terms which are defined in Clause 2. These
terms are printed in italics upon first usage in this Clause.
There are three elements specified in this part of ISO/TR 19566:
a) An encoder is an embodiment of an encoding process. An encoder takes as input digital source
image data and encoder specifications, and by means of a specified set of procedures generates as
output a codestream.
b) A decoder is an embodiment of a decoding process. A decoder takes as input a codestream, and by
means of a specified set of procedures generates as output digital reconstructed image data.
c) The codestream is a compressed image data representation which includes all necessary data to
allow a (full or approximate) reconstruction of the sample values of a digital image. Additional data
might be required that define the interpretation of the sample data, such as colour space or the
spatial dimensions of the samples.
5 Background
5.1 General
Systems designed to work with still images consist of different functionalities ranging from basic
functionalities such as image compression up to image interpretation and search. These functionalities
can be grouped into a layered architecture as exemplified in Figure 1.
8 PROOF/ÉPREUVE © ISO/IEC 2016 – All rights reserved

---------------------- Page: 13 ----------------------
ISO/IEC TR 19566-1:2016(E)

Figure 1 — Layered architecture of the JPEG Systems
Layer 1 is the basic layer and groups different coding algorithms and metadata schemes that allow
representing picture data and descriptive information.
Layer 2 groups the different file formats that embed one or multiple codestreams emitted by the image
codecs. By these means, it is possible to specify supplemental information, such as the represented
colour space, or provide means for compositing multiple images.
Layer 3 defines functionality extensions that enhance the basic functionalities of the lower layers.
Corresponding examples are, for instance, privacy protection or error robustness. Such extensions
might operate on the codestream layer or the file format layer.
Layer 4 provides means for transport of image data over networks. This comes along with the definition
of appropriate ways of dividing codestreams or file formats into units that can be carried with existing
network protocols.
Layer 5 finally groups services operating with images, together with associated APIs. Examples are, for
instance, facilities for image search.
This part of ISO/IEC 19566-1 is about Layer 1 and Layer 2 of the architecture depicted in Figure 1.
© ISO/IEC 2016 – All rights reserved PROOF/ÉPREUVE 9

---------------------- Page: 14 ----------------------
ISO/IEC TR 19566-1:2016(E)

Annex A lays out the general strategies for storage of information.
Annex B explains the preferred syntax for organization of information, while Annex C derives resulting
examples and guidelines for the format of a compressed file.
Annex D then explains a more compact form of information storage.
Annex E finally lists a set of functionalities that are typically to be exposed on the file format level.
5.2 Organization of information
In the vast majority of cases, data as considered in JPEG standards can be hierarchically structured.
An image, for instance, consists of descriptive data such as its extensions, number of colours, etc. This
block of descriptive data is followed by the individual pixels. These pixels can be grouped into tiles,
subbands and blocks or some similar structure. In addition, metadata, such as the colour space used to
encode the information, might be associated with the image.
Consequently, these data are typically stored in hierarchical data structures having the form of lists,
trees, tables or similar concepts. It is the task of every International Standard describing a compression
format to define its own syntax in order to map such a hierarchical data structure into a linear sequence
of bytes as they can be stored by today’s electronic devices. Such an approach permits to optimally
adjust the way how the data are coded to the desired needs.
On the other hand, this comes along with the drawback of a multitude of differently looking storage
and coding schemes that are difficult to handle in a harmonized manner. Applications that need to
decompose a junk of data into its pieces, and supporting different file or codestream formats, need to
implement multiple mechanisms for identifying the desired subpieces. Moreover, it is difficult to reuse
software and hardware components when implementing a standard because each of them uses its
own approach to organize the data. Finally, general operations such as cropping a resolution are only
possible when understanding the coded file in all its detail, even when the data will not be decoded by
the application.
Consequently, this part of ISO/TR 19566 lays out guidelines for harmonized storage of image and image
related data covering layers 1 and 2 of the JPEG Systems.
10 PROOF/ÉPREUVE © ISO/IEC 2016 – All rights reserved

---------------------- Page: 15 ----------------------
ISO/IEC TR 19566-1:2016(E)

Annex A
(informative)

Organization of information — Signalling, packaging and storage
A.1 General
This Annex gives an overview to the organization of codestreams and files containing coded images.
A.2 Storage layers
Storage of information takes place on two different layers, namely the codestream layer and the file
format layer. The codestream layer comprises all the data that are necessary to transform a coded
representation into a sequence of pixels. Corresponding examples include image extensions, number of
colour components and the coded data itself. The file format layer provides supplementary information
that help to place the data into the correct context, such as colour spaces or composition rules.
Information that is stored on the file format level includes the following:
— colour spaces and profiles (such as ICC) defining the colour space in which the pixels are to be
interpreted;
— thumbnail information;
— composition and animation information;
— annotations of regions of interest;
— EXIF data;
— XML data;
— information on resolutions (physical information);
— translation from pixel domain to presentation on physical devices;
— vendor-specific information;
— descriptive data (for search);
— other metadata.
A.3 Interface aspects
File formats and codestream formats offer an interface that can be used by the other layers of the
Systems framework shown in Figure 1. This interface covers two aspects:
— The semantics define what functionalities are offered by the codestream or file format, such as the
capability to extract a lower resolution.
— The syntax describes how this functionality can be accessed. For instance, it describes the syntax of
the box that covers the low resolution representation and how it can be located in the file. A client
thus knowing the syntax is hence capable of accessing the corresponding information.
© ISO/IEC 2016 – All rights reserved PROOF/ÉPREUVE 11

---------------------- Page: 16 ----------------------
ISO/IEC TR 19566-1:2016(E)

A.4 Interoperability
In order to be able to create an interoperable Systems framework, it is desirable to have a common
interface on the file format and codestream levels. Otherwise, every Systems tool, such as error
protection, needs to be specifically adapted to every file format and the contained codestream. Ideally,
both syntax and semantics are compatible on both the codestream and the file format layer.
Unfortunately, this ideal situation cannot be reached in all cases because of several reasons:
— legacy codecs might not be compliant to the selected interface definition;
— codecs or file formats might be designed in such a way that they cannot offer a certain functionality
in order to limit complexity or to be specifically adapted to certain applications.
The Systems framework thus needs to be constructed as shown in Figure A.1. Ideally, both common
syntax and semantics are employed. In case file formats or codestreams share the same semantics, but
use different syntax, a syntax adaptation needs to take place. This might be by adapting the desired
Systems functionality directly to the syntax or semantics, or by first translating the syntax into an
alternative format before applying the desired Systems functionality. In case both semantics and syntax
differs, only a subset of the JPEG Systems functionality may be provided for the codestream or file format.
Figure A.1 — File format interface for the Systems framework
A.5 Syntax elements for information signalling, transport and storage
Carriage of information needs to be signalled on both the codestream and the file format level. For both
of them, different syntax descriptions have been elaborated in the past.
On the codestream level, APP markers are a simple and lightweight mechanism in order to carry
information. During the continued development of the standards, it became, however, obvious that they
are rather difficult to extend in a consistent manner.
The box-based structure is a much more generic approach that has its origins in different file formats
...