ISO/IEC 19795-6:2012

INTERNATIONAL ISO/IEC
STANDARD 19795-6
First edition
2012-02-01
Information technology — Biometric
performance testing and reporting —
Part 6:
Testing methodologies for operational
evaluation
Technologies de l'information — Essais et rapports de performances
biométriques —
Partie 6: Méthodologies d'essai pour l'évaluation opérationnelle

Reference number
©
ISO/IEC 2012
©  ISO/IEC 2012
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means,
electronic or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the address below or
ISO's member body in the country of the requester.
ISO copyright office
Case postale 56  CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 749 09 47
E-mail copyright@iso.org
Web www.iso.org
Published in Switzerland
ii © ISO/IEC 2012 – All rights reserved

Contents Page
Foreword . v
Introduction . vi
1  Scope . 1
2  Conformance . 1
3  Normative references . 1
4  Terms and definitions . 2
5  Operational evaluation overview . 3
5.1  Operational evaluation goals . 3
5.2  Operational performance metrics . 4
5.3  Operational evaluation methods . 4
5.4  Determining operational performance . 4
5.5  Use of technology and scenario evaluation methodologies in evaluating operational
systems . 5
6  Operational evaluation . 5
6.1  Purpose and scope . 5
6.1.1  General . 5
6.1.2  Criteria for system inclusion . 5
6.1.3  System specification . 5
6.1.4  Biometric functionality . 6
6.1.5  Performance measures . 6
6.2  Application characteristics . 6
6.2.1  General . 6
6.2.2  Concept of operations . 7
6.2.3  Guidance and instruction . 7
6.2.4  Levels of effort and decision policies . 8
6.2.5  Multiple-instance systems . 8
6.2.6  Environment . 9
6.2.7  Deployment factors . 9
6.2.8  Acclimatization . 10
6.2.9  Habituation . 10
6.3  Test Plan . 10
6.3.1  General . 10
6.3.2  System implementation and configuration. 11
6.3.3  Test population . 11
6.3.4  Test transactions . 12
6.4  Performance measurement . 14
6.4.1  Throughput . 14
6.4.2  Enrolment analysis . 15
6.4.3  Recognition analysis . 15
6.5  Reporting . 16
6.5.1  Reporting planned test results . 16
6.5.2  Reporting additional analyses . 16
6.5.3  Reporting observations . 17
6.5.4  Report structure . 17
6.6  Record keeping . 17
Annex A (informative) Non-mandatory performance metrics and reporting . 18
Annex B (informative) Sub-transaction events in operational testing . 20
© ISO/IEC 2012 – All rights reserved iii

Annex C (informative) Sample operational test specification .21
Annex D (informative) Methods to determine test size .23
Annex E (informative) Operational system monitoring .25
Annex F (informative) Operational habituation testing .27
Annex G (informative) Sample operational test report outline .28
Bibliography .30

iv © ISO/IEC 2012 – All rights reserved

Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are members of
ISO or IEC participate in the development of International Standards through technical committees
established by the respective organization to deal with particular fields of technical activity. ISO and IEC
technical committees collaborate in fields of mutual interest. Other international organizations, governmental
and non-governmental, in liaison with ISO and IEC, also take part in the work. In the field of information
technology, ISO and IEC have established a joint technical committee, ISO/IEC JTC 1.
International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2.
The main task of the joint technical committee is to prepare International Standards. Draft International
Standards adopted by the joint technical committee are circulated to national bodies for voting. Publication as
an International Standard requires approval by at least 75 % of the national bodies casting a vote.
Attention is drawn to the possibility that some of the elements of this document may be the subject of patent
rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights.
ISO/IEC 19795-6 was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 37, Biometrics.
ISO/IEC 19795 consists of the following parts, under the general title Information technology — Biometric
performance testing and reporting:
 Part 1: Principles and framework
 Part 2: Testing methodologies for technology and scenario evaluation
 Part 3: Modality-specific testing [Technical Report]
 Part 4: Interoperability performance testing
 Part 5: Access control scenario and grading scheme
 Part 6: Testing methodologies for operational evaluation
 Part 7: Testing of on-card biometric comparison algorithms
© ISO/IEC 2012 – All rights reserved v

Introduction
Operational tests evaluate complete biometric systems in the targeted operational environment with the target
population. Tests may encompass performance monitoring of operational systems or assessment of
performance in operational trials.
Operational performance assessment may be based on:
 data collected by the operational system in the course of normal operation;
 additional data collected during normal system use, but with the system running in an “evaluation mode”
allowing extra data to be collected;
 data collected with a set of test subjects considered separately from the subject base of the operational
system.
Operational evaluation differs from technology or scenario evaluation in that the subject base, environment,
and system design are no longer controlled for the purpose of repeatable testing, but vary in accordance with
operational use. Examples of uncontrolled variables include the legitimacy of the subject’s identity claim,
environmental effects from weather or lighting, or the variability of system use by different individuals.
The overarching goals of operational testing are to measure or monitor operational biometric system
performance over a period of time.
Subgoals of operational testing may include:
 to determine if performance meets the requirements specified for a particular application or the claims
asserted by the supplier;
 to determine the need to adjust or configure the system to improve performance;
 to predict performance as the numbers of subjects, locations, or devices increase;
 to obtain information on the target population and environmental parameters found to affect system
performance;
 to obtain performance data from a pilot implementation;
 to obtain performance data to benchmark future systems.
This part of ISO/IEC 19795 provides the test planning, test conduct, performance measurement, test reporting,
and record keeping requirements to be followed during a biometric system’s operational evaluation.
vi © ISO/IEC 2012 – All rights reserved

INTERNATIONAL STANDARD ISO/IEC 19795-6:2012(E)

Information technology — Biometric performance testing and
reporting —
Part 6:
Testing methodologies for operational evaluation
1 Scope
This part of ISO/IEC 19795:
 provides guidance on the operational testing of biometric systems;
 specifies performance metrics for operational systems;
 details data that may be retained by operational systems to enable performance monitoring; and
 specifies requirements on test methods, recording of data, and reporting of results of operational
evaluations.
NOTE Some operational biometric systems perform a single biometric function. For example, in the initial stages of
rollout of biometric passports, the operational system might be performing biometric enrolment only. Operational
evaluation of such systems is within the scope of this part of ISO/IEC 19795.
This part of ISO/IEC 19795 does not:
 cover testing of operational systems in the laboratory or
 address vulnerability testing.
2 Conformance
An operational evaluation is in conformance with this part of ISO/IEC 19795 if it is planned, executed and
reported in accordance with the requirements of Clause 6.
3 Normative references
The following referenced documents are
...