ISO/IEC TR 29794-5:2010

TECHNICAL ISO/IEC
REPORT TR
29794-5
First edition
2010-04-01


Information technology — Biometric
sample quality —
Part 5:
Face image data
Technologies de l'information — Qualité d'échantillon biométrique —
Partie 5: Données d'image de face




Reference number
ISO/IEC TR 29794-5:2010(E)
©
ISO/IEC 2010

---------------------- Page: 1 ----------------------
ISO/IEC TR 29794-5:2010(E)
PDF disclaimer
This PDF file may contain embedded typefaces. In accordance with Adobe's licensing policy, this file may be printed or viewed but
shall not be edited unless the typefaces which are embedded are licensed to and installed on the computer performing the editing. In
downloading this file, parties accept therein the responsibility of not infringing Adobe's licensing policy. The ISO Central Secretariat
accepts no liability in this area.
Adobe is a trademark of Adobe Systems Incorporated.
Details of the software products used to create this PDF file can be found in the General Info relative to the file; the PDF-creation
parameters were optimized for printing. Every care has been taken to ensure that the file is suitable for use by ISO member bodies. In
the unlikely event that a problem relating to it is found, please inform the Central Secretariat at the address given below.


COPYRIGHT PROTECTED DOCUMENT


©  ISO/IEC 2010
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means,
electronic or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the address below or
ISO's member body in the country of the requester.
ISO copyright office
Case postale 56 • CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 749 09 47
E-mail copyright@iso.org
Web www.iso.org
Published in Switzerland

ii © ISO/IEC 2010 – All rights reserved

---------------------- Page: 2 ----------------------
ISO/IEC TR 29794-5:2010(E)
Contents Page
Foreword .iv
Introduction.v
1 Scope.1
2 Normative references.1
3 Terms and definitions .1
4 Abbreviated terms.1
5 Approaches to Face Image Quality .2
6 Categorization of Facial Quality.2
7 Facial Image Quality Analysis.4
7.1 Dynamic Subject Characteristics .5
7.1.1 Subject’s Behaviour .5
7.1.2 Analysis Based on Statistical Differences of the Left and Right Half of the Face.5
7.2 Static Characteristics of the Acquisition Process .7
7.2.1 Image Resolution and Size.8
7.2.2 Noise.8
7.3 Characteristics of Image Acquisition .8
7.3.1 Image Properties.8
7.3.2 Image Appearance.9
7.3.3 Illumination Intensity.9
7.3.4 Image Brightness.9
7.3.5 Image Contrast.10
7.3.6 Exposure.11
7.3.7 Focus, Blur and Sharpness.11
7.3.8 Colour.12
7.3.9 Subject-Camera distance.12
Bibliography.13

© ISO/IEC 2010 – All rights reserved iii

---------------------- Page: 3 ----------------------
ISO/IEC TR 29794-5:2010(E)
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are members of
ISO or IEC participate in the development of International Standards through technical committees
established by the respective organization to deal with particular fields of technical activity. ISO and IEC
technical committees collaborate in fields of mutual interest. Other international organizations, governmental
and non-governmental, in liaison with ISO and IEC, also take part in the work. In the field of information
technology, ISO and IEC have established a joint technical committee, ISO/IEC JTC 1.
International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2.
The main task of the joint technical committee is to prepare International Standards. Draft International
Standards adopted by the joint technical committee are circulated to national bodies for voting. Publication as
an International Standard requires approval by at least 75 % of the national bodies casting a vote.
In exceptional circumstances, the joint technical committee may propose the publication of a Technical Report
of one of the following types:
⎯ type 1, when the required support cannot be obtained for the publication of an International Standard,
despite repeated efforts;
⎯ type 2, when the subject is still under technical development or where for any other reason there is the
future but not immediate possibility of an agreement on an International Standard;
⎯ type 3, when the joint technical committee has collected data of a different kind from that which is
normally published as an International Standard (“state of the art”, for example).
Technical Reports of types 1 and 2 are subject to review within three years of publication, to decide whether
they can be transformed into International Standards. Technical Reports of type 3 do not necessarily have to
be reviewed until the data they provide are considered to be no longer valid or useful.
Attention is drawn to the possibility that some of the elements of this document may be the subject of patent
rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights.
ISO/IEC TR 29794-5, which is a Technical Report of type 2, was prepared by Joint Technical Committee
ISO/IEC JTC 1, Information technology, Subcommittee SC 37, Biometrics.
ISO/IEC 29794 consists of the following parts, under the general title Information technology — Biometric
sample quality:
⎯ Part 1: Framework
⎯ Part 4: Finger image data [Technical Report]
⎯ Part 5: Face image data [Technical Report]
iv © ISO/IEC 2010 – All rights reserved

---------------------- Page: 4 ----------------------
ISO/IEC TR 29794-5:2010(E)
Introduction
The purpose of this part of ISO/IEC 29794 is to define and specify methodologies for computation of objective,
quantitative quality scores for facial images. Furthermore, the purpose, intent, and interpretation of face quality
scores are defined.
ISO/IEC 19794-5, Information technology — Biometric data interchange formats — Part 5: Face image data,
already gives some specifications that are related to
⎯ scene constraints of the facial images,
⎯ photographic properties of the facial images, and
⎯ digital image attributes of the facial images.
Within this part of ISO/IEC 29794, a sample of a classification scheme of facial quality is exemplified and
approaches for the determination of certain aspects of quality are introduced.

© ISO/IEC 2010 – All rights reserved v

---------------------- Page: 5 ----------------------
TECHNICAL REPORT ISO/IEC TR 29794-5:2010(E)

Information technology — Biometric sample quality —
Part 5:
Face image data
1 Scope
For aspects of quality specific to facial images, this part of ISO/IEC 29794:
⎯ specifies terms and definitions that are useful in the specification, use and testing of face image quality
metrics;
⎯ defines the purpose, intent, and interpretation of face image quality scores.
Performance assessment of quality algorithms and standardization of quality algorithms are outside the scope
of this part of ISO/IEC 29794.
2 Normative references
The following referenced documents are indispensable for the application of this document. For dated
references, only the edition cited applies. For undated references, the latest edition of the referenced
document (including any amendments) applies.
ISO/IEC 29794-1, Information technology — Biometric sample quality — Part 1: Framework
3 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO/IEC 29794-1 and the following apply.
3.1
comparison score
numerical value (or set of values) resulting from a comparison
3.2
face quality assessment algorithm
algorithm that computes a quality score for a given face image sample
3.3
facial image
electronic image-based representation of the portrait of a person
4 Abbreviated terms
CCD Charge-coupled device
DCT Discrete Cosine Transform
© ISO/IEC 2010 – All rights reserved 1

---------------------- Page: 6 ----------------------
ISO/IEC TR 29794-5:2010(E)
GCF Global Contrast Factor
FQAA Face Quality Assessment Algorithm
QS Quality Score
FQS Face Quality Score
QSN Quality Score Normalization
5 Approaches to Face Image Quality
Face Image Quality can be defined in many ways, depending on the application. For the purpose of this part
of ISO/IEC 29794 standard Face Image Quality is defined in relation to the use of facial images with
automated face recognition systems. The performance of an automated face recognition system is affected by
the amount of defect or the degree of imperfection present in the face image. The knowledge of quality can,
and is currently being used to, process face images differently, by either invoking some image enhancement
or normalization methods prior to feature extraction, invoking different matchers based on quality, or simply
changing the threshold. The use of face image quality assessment to enhance the overall performance of the
system is increasing [3, 4, 5].
A very important application of real-time quality analysis of faces is Face Recognition in Video, also referred to
as Face in a Crowd, Recognition on the move, or Face at a Distance, e.g [21].
This part of ISO/IEC 29794 shows some approaches for estimating Face Image Quality. The aim is to give the
reader examples of assessment algorithms. Note, that these algorithms have pros and cons and no one
algorithm is likely to be suitable for all facial images. Standardization of these algorithms is out of scope of this
part of ISO/IEC 29794.
The following related work is being done in ISO/IEC JTC1 SC37 [1, 2]:
• ISO/IEC 29794-1 suggests the use of Quality Algorithm Identification (QAID), or Quality Score Percentile
Rank upon standardization of a Quality Score Normalization Dataset (QSND).
This part of ISO/IEC 29794 adopts the following approach for face sample quality description:
• Specifying characterization of the facial quality and possible defects of face biometric samples in
categorized aspects.
• Showing how FQAAs can be used to derive face quality scores (FQSs) related to specific characteristics
and associated possible defects. An FQAA typically analyzes a face sample locally at the pixel or feature
level and fuses the local analysis results over a global region. An FQS evaluates one or more
characteristics and associated potential defects, and provides an indicator of the quality.
A typical approach of a system for generation of quality scores for facial images then takes the atomic FQSs
generated by the FQAAs and combines them to a final quality score. The final quality score must predict
performance metrics such as either false match or false non-match of an automatic facial image recognition.
6 Categorization of Facial Quality
Different factors affect the quality of the facial image with respect to biometric systems’ performance. A
successful recognition will be based on the biometric characteristics of the subject and a number of factors
that influence these characteristics such as variations (e.g. due to ageing) and the environmental conditions in
the acquisition process:
• Influence of subjects characteristics on biometric performance,
• Influence of the acquisition process (including the capturing device) on biometric performance.
2 © ISO/IEC 2010 – All rights reserved

---------------------- Page: 7 ----------------------
ISO/IEC TR 29794-5:2010(E)
This classification is not sufficient, as it does not distinguish between static and dynamic characteristics and
properties:
• static subjects characteristics are related to anatomical characteristics of the subject,
• dynamic subject characteristics are related to subjects behaviour during the acquisition process,
• static properties of the acquisition process are related to physical properties of the capturing device and
effects caused by the sample processing chain,
• dynamic properties of the acquisition process are related to environmental conditions during the capturing
process.
Table 1 shows a classification scheme that differentiates between the dynamic versus static properties as well
as the subject versus the acquisition process characteristics affecting facial quality.
Table 1 — Characterization of Facial Quality
Subject characteristics Acquisition process
Biological characteristics, like Acquisition process and capture device properties, like
- anatomical characteristics (e.g. head - image enhancement and data reduction process
dimensions, eye positions)
- physical properties (e.g. image resolution and
- injuries and scars contrast)
- ethnic group - optical distortions
- impairment - static properties of the background, e.g.
Static
wallpaper

- camera characteristics
Other static characteristics
o sensor resolution
- Heavy facial wears, such as thick or
dark glasses - scene characteristics
- Makeup o geometric distortion
- Permanent jewellery
Subject characteristics and behaviour, like Scenery, like
- closed eyes - dynamic characteristics of the background like
moving objects
- (exaggerated) expression
- variation in lightning and related potential defects
- hair across the eye
as
- head pose
o deviation from the symmetric lighting
- subject posing (frontal / non frontal to
o uneven lighting on the face area
camera)
o Extreme strong or weak illumination

- subject posing , e.g.
Dynamic
o too far (face too small), or too near (face
too big)
o out of focus (low sharpness)
o partial occlusion of the face
- Acquisition process and capture device
properties, such as
o camera characteristics
o dynamic range (response to weak and
strong lighting)
© ISO/IEC 2010 – All rights reserved 3

---------------------- Page: 8 ----------------------
ISO/IEC TR 29794-5:2010(E)
The classification scheme (as all other content of this part of ISO/IEC 29794) is given for informative purposes
only. The proposed classification scheme is certainly not the only possible scheme, but it is very useful since it
separates design from character, i.e. it can be used to guide quality by design and hence performance
improvement.
This characterization and the related categories of defects, degradations and interferences affect the
performance of an automated facial recognition system. What is not considered in this part of ISO/IEC 29794
are the effects of printing on the given facial images (e.g. in a passport production process), which likely
introduces further distortions especially with respect to image appearance and noise.
7 Facial Image Quality Analysis
Different aspects have to be considered in a facial image quality analysis. Some of them are already defined
in related standardization documents. Different categories can be identified:
1. image properties like the size of the image or its resolution,
2. image appearance characteristics like the exposure or noise,
3. scenery characteristics like lighting or background,
4. characteristics like the consistency between the skin colour on the image and the skin colour of the
subject,
5. the behaviour of the subject.
For some of these properties and characteristics metrics already exist. Some properties and characteristics,
however, are much harder to be assessed and evaluated like the consistency of the skin colour on the image
and the skin colour of the subject.
Furthermore, for some properties and characteristics, like the eye distance (in pixels), requirements are
defined in ISO/IEC 19794-5 [37]. Their evaluation requires more complex algorithms and technologies from
computer vision and image understanding. Therefore, a simple metric can not be given without considering
the implementation that is needed to extract the corresponding features. In addition to this, different core
concepts might be possible, e.g. different principles exist to automatically determine the eye positions in facial
images. It may be possible to derive normalized quality scores as described in ISO/IEC 29794-1 (QSND). For
some metrics, the variation between the enrolled images and that of the query images plays a bigger role in
predicting performance than does the absolute metric applied to a single image. For instance [38] shows that
performance is more affected by the relationship between the resolution of enrolled images and the query
images than by absolute measure of resolution applied to each of them.
An FQAA can examine the image without a segmentation of the facial area (e.g. to assess static
characteristics of the acquisition process like the compression rate and resulting compression artefacts,
sensor resolution when measuring the size of an image) or perform an analysis on the facial area only (e.g.
when estimating the pose of a subject). Local structures of a face may be defined by pixel values (raw or
processed) within local regions; they may be fused globally to give a single quality score. Various FQAA can
be developed, for different quality aspects related to environment, of camera, and/or subject showing different
performance on different data sets. It is out of scope of this part of ISO/IEC 29794 to rate or rank the different
approaches.
For some of the quality measures it is assumed that the face has been detected, and the facial area is
normalized properly in geometry according to some landmarks such as the eye positions. Only the cropped
face region is used for the analysis in this case.
4 © ISO/IEC 2010 – All rights reserved

---------------------- Page: 9 ----------------------
ISO/IEC TR 29794-5:2010(E)
7.1 Dynamic Subject Characteristics
7.1.1 Subject’s Behaviour
Typical characteristics that are related to the subject’s behaviour include:
• closed or open eyes,
• closed or open mouth,
• any kind of expression, e.g. smiling or neutral,
• head pose, e.g. frontal or rotated in any direction.
Similar to the scenery properties or the characteristics, the quantification of these parameters requires the
recognition of background, faces and facial characteristics.
Again, different core algorithms can be implemented and their performance values can be used. A reduction in
complexity can be achieved by selecting algorithms or concepts that are most commonly applied if this
information is available.
7.1.2 Analysis Based on Statistical Differences of the Left and Right Half of the Face
7.1.2.1 Lighting Symmetry
The following approaches are based on the assumption that the images being analyzed are 2D portrait style
images such as those specified in ISO/IEC 19794-5:2005/Amd.1, Information technology — Biometric data
interchange formats — Face image data — Amendment 1: Conditions for taking photographs for face image
data. This relates to facial and environment semantics. Left-right symmetry can be used to evaluate quality of
lighting and pose [36]. The face region is divided into left and right halves at the mid-line of the eyes
(Figure 1).The symmetry analysis below examines differences between the corresponding left-right locations.
The difference value indicates the degree of asymmetry in some local image properties, e.g., raw pixel value,
or locally-filtered pixels value. The local image filter can be Gabor filter [9,10], Local Binary Pattern (LBP) filter
[11-13], Ordinal filter [14-17], or any other suitable local filter. The left-right difference value provides a quality
score for the lighting (i.e. how symmetric the lighting is), or the pose quality (i.e. how frontal the pose is).
Although the majority of faces seem to be left –right symmetric some faces could have significant deviation,
e.g. caused by marks, discoloration etc. that would affect symmetry based quality analysis metrics Images are
taken from Yale face image database [18].




Æ
Figure 1 — Division of a face into left and right half regions at the mid-line of the eyes

L R
The difference can be based on histograms H and H of some local features in the left and right half
mn∗ mn∗
regions where m is the dimensionality of the feature vector, and n is the number of bins in the histogram. A
histogram difference can be calculated as follows:
LR
DH=−H
im∗∗n mn
© ISO/IEC 2010 – All rights reserved 5

---------------------- Page: 10 ----------------------
ISO/IEC TR 29794-5:2010(E)
where the metric |.| is some suitable form of histogram distance, e.g., histogram intersection, cross-entropy,
Kullback–Leibler divergence. The larger the difference value is, the less left-right symmetric the face image is,
and the lower the image quality is in some aspect.
One possibility is using image normalized pixel values. The following presents an example of an FQAA for
lighting symmetry:
(1) Normalize the range of pixel values in the cropped face region using a suitable normalization or
equalization algorithm.
(2) Calculate the difference between normalized values for each pixel pair of sub windows at left-right mirror
2 locations.
(3) Calculate a suitable sum of the abstract values of the differences.
The sum is a metric of lighting asymmetry. The larger the sum value is, the less the left-right symmetric the
face image is, and the lower the image quality is in terms of the lighting symmetry.
Figure 2 shows the lighting asymmetry for two face images. 2300 pairs of pixels are randomly selected across
an image. The horizontal axis indexes the pair number. The vertical axis corresponds to the lighting
asymmetry. The frontal lighted face image (the one on the left) has lower lighting asymmetry (the darker
curve), whereas the sided lighted one has higher lighting asymmetry (lighter curve). Figure 3 shows another
example, where the lighting is even more asymmetric. As can be seen, the symmetrically lighted face has
much lower difference values.


Figure 2 — A result of asymmetric lighting, and the distributions of asymmetry


Figure 3 — Another result of asymmetric lighting, and the distributions of asymmetry
6 © ISO/IEC 2010 – All rights reserved

---------------------- Page: 11 ----------------------
ISO/IEC TR 29794-5:2010(E)
Note, that this method only works for frontal or almost frontal images. On the other hand, if we assume that
asymmetry of the object is small in comparison to the other factors we can use this to evaluate quality of
lighting, to the extent that the image symmetry mismatch is due to these conditions
7.1.2.2 Pose Symmetry
This should be done based on pose-sensitive image properties. One possibility is using Local Binary Pattern
(LBP) filtered pixel values. The following presents an example of an FQAA for pose symmetry:
(1) Perform filtering using LBP filters.
(2) Calculate the difference between filtered values for each pixel pair of sub windows at left-right mirror
locations.
(3) Calculate a suitable sum of the absolute values of the differences.
The sum is a metric of pose asymmetry. The larger the sum value is the more the face is rotated left or right
and the lower the image quality is in terms of pose symmetry.
Figure 4 gives an example of the pose asymmetric values for 4 pose categories (different curves) and
10 people (horizontal axis) of the local differences for the following two face images (one with symmetric
lighting and the other not). The differences are calculated between pairs of pixels at 2300 random locations.
The means of the differences are plotted. From bottom to top, the curves correspond to the four pose
categories from left to right. The curves (from bottom to top) correspond to the face asymmetry values for the
four pose categories from left to right.

Figure 4 — Result of pose asymmetry
Note, that from the outcome of the described method one cannot distinguish between asymmetry of the face
and/or asymmetry of the illumination and/or pose. On the other hand, if we assume that asymmetry of the
object is small in comparison to the other factors we can use this to evaluate quality of lighting and pose, to
the extent that the image symmetry mismatch is due to these conditions.
7.2 Static Characteristics of the Acquisition Process
Typical scenery characteristics describing the environmental influence are
• image enhancement and data reduction process, i.e. image resolution and size,
• static camera characteristics like resolution,
• static properties of the background like wallpapers.
Depending on the property or the characteristic, the quantification of these parameters requires the
recognition of background, faces and facial characteristics.
Here, different core algorithms can be implemented and their performance values can be used. A reduction of
the complexity can be achieved by selection of the algorithms or concepts that are applied in the most
significant recognition system if this information is available.
© ISO/IEC 2010 – All rights reserved 7

---------------------- Page: 12 ----------------------
ISO/IEC TR 29794-5:2010(E)
7.2.1 Image Resolution and Size
An estimation of the size or the resolution is difficult as important parameters during the image acquisition
process cannot be estimated. As an alternative, the numbers of row and column pixels can be used as an
indication of nominal resolution for a standard subject-to-camera geometry. More meaningful measures of
image resolvability may be made once the head and face are segmented from the image background and
particularly once eye coordinates are located. Interpupillary distance in pixels provides a measure of pixel
coverage relative to facial features. Moreover, applying a statistical measure of average interpupillary distance,
e.g., 63mm [19], pixel density may be converted to spatial sample rate in, e.g., samples/millimetre.
This does not, however, take into account image processing operations that remove detail information and
therefore reduce the resolution of the image. Among these operations are
• low-pass filtering and high-frequency noise removal,
• a down-sampling process that is followed by an up-sampling process.
7.2.2 Noise
The noise in facial images depends on the different processes that are required to result in a digital image.
The introduced noise is specific according to the device or process involved. Relevant noise sources include
• digital image acquisition devices, e.g. the image sensor of a digital camera,
• analogue image acquisition devices,
• image scanning devices,
• image compression algorithms, e.g. JPEG or Wavelet compression.
7.2.2.1 Image Acquisition Noise
Liu et al. [22] present a method to estimate an upper bound for the CCD acquisition noise. The authors
estimate this upper bound based on a piecewise
...