ISO/IEC 19795-6:2012

INTERNATIONAL ISO/IEC
STANDARD 19795-6
First edition
2012-02-01
Information technology — Biometric
performance testing and reporting —
Part 6:
Testing methodologies for operational
evaluation
Technologies de l'information — Essais et rapports de performances
biométriques —
Partie 6: Méthodologies d'essai pour l'évaluation opérationnelle

Reference number
©
ISO/IEC 2012
©  ISO/IEC 2012
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means,
electronic or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the address below or
ISO's member body in the country of the requester.
ISO copyright office
Case postale 56  CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 749 09 47
E-mail copyright@iso.org
Web www.iso.org
Published in Switzerland
ii © ISO/IEC 2012 – All rights reserved

Contents Page
Foreword . v
Introduction . vi
1  Scope . 1
2  Conformance . 1
3  Normative references . 1
4  Terms and definitions . 2
5  Operational evaluation overview . 3
5.1  Operational evaluation goals . 3
5.2  Operational performance metrics . 4
5.3  Operational evaluation methods . 4
5.4  Determining operational performance . 4
5.5  Use of technology and scenario evaluation methodologies in evaluating operational
systems . 5
6  Operational evaluation . 5
6.1  Purpose and scope . 5
6.1.1  General . 5
6.1.2  Criteria for system inclusion . 5
6.1.3  System specification . 5
6.1.4  Biometric functionality . 6
6.1.5  Performance measures . 6
6.2  Application characteristics . 6
6.2.1  General . 6
6.2.2  Concept of operations . 7
6.2.3  Guidance and instruction . 7
6.2.4  Levels of effort and decision policies . 8
6.2.5  Multiple-instance systems . 8
6.2.6  Environment . 9
6.2.7  Deployment factors . 9
6.2.8  Acclimatization . 10
6.2.9  Habituation . 10
6.3  Test Plan . 10
6.3.1  General . 10
6.3.2  System implementation and configuration. 11
6.3.3  Test population . 11
6.3.4  Test transactions . 12
6.4  Performance measurement . 14
6.4.1  Throughput . 14
6.4.2  Enrolment analysis . 15
6.4.3  Recognition analysis . 15
6.5  Reporting . 16
6.5.1  Reporting planned test results . 16
6.5.2  Reporting additional analyses . 16
6.5.3  Reporting observations . 17
6.5.4  Report structure . 17
6.6  Record keeping . 17
Annex A (informative) Non-mandatory performance metrics and reporting . 18
Annex B (informative) Sub-transaction events in operational testing . 20
© ISO/IEC 2012 – All rights reserved iii

Annex C (informative) Sample operational test specification .21
Annex D (informative) Methods to determine test size .23
Annex E (informative) Operational system monitoring .25
Annex F (informative) Operational habituation testing .27
Annex G (informative) Sample operational test report outline .28
Bibliography .30

iv © ISO/IEC 2012 – All rights reserved

Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are members of
ISO or IEC participate in the development of International Standards through technical committees
established by the respective organization to deal with particular fields of technical activity. ISO and IEC
technical committees collaborate in fields of mutual interest. Other international organizations, governmental
and non-governmental, in liaison with ISO and IEC, also take part in the work. In the field of information
technology, ISO and IEC have established a joint technical committee, ISO/IEC JTC 1.
International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2.
The main task of the joint technical committee is to prepare International Standards. Draft International
Standards adopted by the joint technical committee are circulated to national bodies for voting. Publication as
an International Standard requires approval by at least 75 % of the national bodies casting a vote.
Attention is drawn to the possibility that some of the elements of this document may be the subject of patent
rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights.
ISO/IEC 19795-6 was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 37, Biometrics.
ISO/IEC 19795 consists of the following parts, under the general title Information technology — Biometric
performance testing and reporting:
 Part 1: Principles and framework
 Part 2: Testing methodologies for technology and scenario evaluation
 Part 3: Modality-specific testing [Technical Report]
 Part 4: Interoperability performance testing
 Part 5: Access control scenario and grading scheme
 Part 6: Testing methodologies for operational evaluation
 Part 7: Testing of on-card biometric comparison algorithms
© ISO/IEC 2012 – All rights reserved v

Introduction
Operational tests evaluate complete biometric systems in the targeted operational environment with the target
population. Tests may encompass performance monitoring of operational systems or assessment of
performance in operational trials.
Operational performance assessment may be based on:
 data collected by the operational system in the course of normal operation;
 additional data collected during normal system use, but with the system running in an “evaluation mode”
allowing extra data to be collected;
 data collected with a set of test subjects considered separately from the subject base of the operational
system.
Operational evaluation differs from technology or scenario evaluation in that the subject base, environment,
and system design are no longer controlled for the purpose of repeatable testing, but vary in accordance with
operational use. Examples of uncontrolled variables include the legitimacy of the subject’s identity claim,
environmental effects from weather or lighting, or the variability of system use by different individuals.
The overarching goals of operational testing are to measure or monitor operational biometric system
performance over a period of time.
Subgoals of operational testing may include:
 to determine if performance meets the requirements specified for a particular application or the claims
asserted by the supplier;
 to determine the need to adjust or configure the system to improve performance;
 to predict performance as the numbers of subjects, locations, or devices increase;
 to obtain information on the target population and environmental parameters found to affect system
performance;
 to obtain performance data from a pilot implementation;
 to obtain performance data to benchmark future systems.
This part of ISO/IEC 19795 provides the test planning, test conduct, performance measurement, test reporting,
and record keeping requirements to be followed during a biometric system’s operational evaluation.
vi © ISO/IEC 2012 – All rights reserved

INTERNATIONAL STANDARD ISO/IEC 19795-6:2012(E)

Information technology — Biometric performance testing and
reporting —
Part 6:
Testing methodologies for operational evaluation
1 Scope
This part of ISO/IEC 19795:
 provides guidance on the operational testing of biometric systems;
 specifies performance metrics for operational systems;
 details data that may be retained by operational systems to enable performance monitoring; and
 specifies requirements on test methods, recording of data, and reporting of results of operational
evaluations.
NOTE Some operational biometric systems perform a single biometric function. For example, in the initial stages of
rollout of biometric passports, the operational system might be performing biometric enrolment only. Operational
evaluation of such systems is within the scope of this part of ISO/IEC 19795.
This part of ISO/IEC 19795 does not:
 cover testing of operational systems in the laboratory or
 address vulnerability testing.
2 Conformance
An operational evaluation is in conformance with this part of ISO/IEC 19795 if it is planned, executed and
reported in accordance with the requirements of Clause 6.
3 Normative references
The following referenced documents are indispensable for the application of this document. For dated
references, only the edition cited applies. For undated references, the latest edition of the referenced
document (including any amendments) applies.
ISO/IEC 19795-1, Information technology — Biometric performance testing and reporting — Part 1: Principles
and framework
© ISO/IEC 2012 – All rights reserved 1

4 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO/IEC 19795-1 and the following apply.
4.1
acclimatization
change, over the course of one or more transactions, of a biometric characteristic that might impact the ability
of a system to process a sample
NOTE Acclimatization is primarily associated with a subject’s temporal adjustment to environmental effects, such as
skin temperature.
4.2
attendant
agent of the biometric system operator who directly interacts with the biometric capture subject
4.3
biometric capture subject
individual who is the subject of a biometric capture process
4.4
biometric data subject
individual whose individualized biometric data is within the biometric system
4.5
biometric probe
biometric data input to an algorithm for comparison to a biometric reference(s)
4.6
biometric operational personnel
individuals, other than the biometric capture subjects, who take an active role in the operation of the biometric
system
NOTE Biometric operational personnel includes biometric system administrators, attendants, and examiners.
4.7
biometric system administrator
person who executes policies and procedures in the administration of a biometric system
4.8
biometric system operator
organization responsible for defining policies and procedures in the operation of a biometric system
4.9
biometric reference
one or more stored biometric samples, biometric templates or biometric models attributed to a biometric data
subject and used for comparison
4.10
comparison attempt limit
maximum allowed number or duration of attempts in a comparison transaction
4.11
enrolment attempt limit
maximum allowed number or duration of attempts in an enrolment transaction
4.12
habituation
familiarity a subject has with the biometric device, system and application
NOTE The level of habituation can affect biometric sample presentation and acquisition device.
2 © ISO/IEC 2012 – All rights reserved

4.13
subject base
set of individuals whose biometric data is intended to be enrolled or compared in operational use of a
biometric system
4.14
system acceptance rate
proportion of recognition transactions in an operational system in which the subject is recognized
NOTE 1 Though the acceptance of an impostor is an incorrect recognition, it can still count as a system acceptance.
NOTE 2 System acceptance rate = 1 – system rejection rate.
4.15
system identification rate
proportion of identification transactions in an operational system in which one or more subjects are identified
4.16
system rejection rate
proportion of recognition transactions in an operational system in which the subject is not recognized
NOTE The system rejection rate differs from the false reject rate in that, in addition to false rejections, it also includes
any rejected impostor transaction and any improper genuine transaction.
4.17
test crew member
selected biometric data subject whose use of the operational system is controlled or monitored as part of the
evaluation
NOTE In an operational evaluation, test subjects can be subjects of the operational system or they can be members
of a test crew using the system specifically for evaluation purposes
5 Operational evaluation overview
5.1 Operational evaluation goals
The overarching goals of operational testing are to measure or monitor operational biometric system
performance.
Subgoals of operational testing may include:
 to determine if performance meets the requirements specified for a particular application or the claims
asserted by the supplier;
 to determine the need to adjust or configure the system to improve performance;
 to predict performance as the numbers of subjects, locations, or devices increase;
 to obtain information on the target population and environmental parameters found to affect system
performance;
 to obtain performance data from a pilot implementation;
 to obtain performance data to benchmark future systems.
Operational evaluation considers the performance of people as well as the equipment, algorithms, and
environment. Consequently, operational testing includes aspects of social science in addition to physical
science, whereas technology testing does not. In general, operational performance will vary over time due to
uncontrolled conditions in people, equipment and environment. For example, if the majority of subjects are
enrolled at the start of operations, with few new enrolments, performance of the system might improve as
© ISO/IEC 2012 – All rights reserved 3

subjects habituate or degrade as subjects’ biometric characteristics age over time away from their enrolled
references.
The performance observed in testing can depend on the operational personnel, such as attendants or
biometric examiners, as well as the biometric subjects. Operational personnel based factors should be taken
into consideration in all aspects of the test from scope definition to reporting (see e.g., references [1] and [2]).
5.2 Operational performance metrics
Recognition metrics for operational testing differ from those used in technology and scenario testing. In
technology and scenario tests, false accept and false reject rates can be measured because the underlying
ground truth is known to the experimenters. Ground truth will generally be unknown in an operational setting
such that an operational test will measure system acceptance and system rejection rates.
Determining the false accept rate and false reject rate from the number of system rejections and acceptances
will require additional observations or controls to determine the legitimacy of identity claims and device
interactions. Similarly, in the case of identification systems, determining identification error rates from the
number of system identifications also requires additional observations or controls.
Performance calculations in technology and scenario tests often exclude rejections in which the subject did
not provide an ideal presentation or did not correctly follow instructions. Operational testing will include such
rejections in measuring the system rejection rate.
5.3 Operational evaluation methods
Operational performance assessment is based on data collected through an operational system. A system
may be configured in “evaluation mode” to collect additional data during normal system use.
Operational performance assessment may be based upon different groups of test subjects:
 data collected with a non-controlled set of test subjects (i.e., a set of test subjects reflective of the subject
base of the operational system);
 data collected with a controlled set of test subjects defined as the “test crew members” (i.e., a set of test
subjects considered and controlled separately from the subject base of the operational system).
If the test crew is specially instructed in their use of the system, evaluation results can be expected to differ
from those encountered operationally. Operational performance can be highly time variant because of
uncontrollable variations in the population, equipment and environment. Variation in performance across these
conditions cannot be predicted.
5.4 Determining operational performance
Performance estimates may be determined from operational data in at least three ways:
 through direct observation of throughput rates, acceptance rates and/or rejection rates;
 through offline computation of throughput and acceptance and rejection rates based on comparison
scores and timing metrics recorded during operations; and
 through offline computation of comparison scores, and acceptance and rejection rates based on samples
acquired during the test and on stored reference data.
Each approach is likely to yield different measures. Further, each approach carries different risks for
miscalculation. For example, recording only direct observations of an access control system does not reveal
whether a rejection was due to a biometric error or an error in the operation of the gate; recording only
comparison scores will not show cases where a reference or probe has been stored or transmitted incorrectly.
4 © ISO/IEC 2012 – All rights reserved

5.5 Use of technology and scenario evaluation methodologies in evaluating operational
systems
In addition to tests based on real operational use of the system, scenario and technology evaluation
(ISO/IEC 19795-2 [3]) can also have a role in determining some aspects of operational performance.
Testing solely in live operation might not be capable of measuring all aspects of operational performance.
Depending on the purpose of the evaluation, certain performance measures might only be determined by
testing for them specifically. Testing in live operation is not meant to guarantee that the system will be
operating under the specific conditions, or with sufficient frequency, in order to draw statistically valid
conclusions. Furthermore, when testing operationally, it might be infeasible to isolate these effects from other
operational factors that also affect performance.
EXAMPLE Environmental factors such as sunlight or humidity can affect sensor performance. Unless the system is
tested to monitor performance in the specific environments, the effects that such factors have on the operational
performance cannot be quantified.
6 Operational evaluation
6.1 Purpose and scope
6.1.1 General
The purpose and scope of the test need to be determined before the test design can be drawn up. The
following elements shall be addressed:
 criteria for system inclusion,
 system specification,
 enrolment and comparison functionality to be evaluated, and
 performance measures of interest.
6.1.2 Criteria for system inclusion
The experimenter shall address the criteria by which biometric system(s) are included in an operational test.
Biometric systems may be included in an operational test due to their having been previously deployed, due to
selection on the part of the biometric system operator, or due to selection on the part of the evaluating entity.
NOTE Testing multiple independent systems could compromise the operational realism of the tests. However, some
elements of the test must be controlled if meaningful comparisons are to be made. Some elements can be controlled
without jeopardizing the operational value of the tests. (See reference [4].)
6.1.3 System specification
Details of the system under test shall be specified as fully as possible. The following elements should be
reported:
 for acquisition devices: manufacturer, model, version, and firmware version as applicable — if the
acquisition device’s core acquisition components are integrated within a third-party device, such as in the
case of a fingerprint sensor incorporated into a peripheral, then manufacturer, model, version, and
firmware of the core acquisition components and those of the peripheral shall be reported;
 for biometric algorithms: provider, version, and revisions, and the values of all field-variable parameters or
settings — biometric algorithms include quality assessment, feature extraction, binning, comparison and
fusion algorithms, and any or all of these might be supplied by different vendors;
© ISO/IEC 2012 – All rights reserved 5

 if the operational system incorporates a biometric application, such as a logical access interface: provider,
title, version, and build of the application;
 for systems tested on or through personal computers, personal digital assistants or other computing
devices: platform, operating systems, processing power, memory, manufacturer, and model of computing
device;
 details of system architecture and data flow between biometric data acquisition, processing, and storage
components;
 data flow between system components;
 system configuration (e.g., in the case reference updating, does the system use a single biometric
reference for all subsequent comparison attempts or is the biometric reference updated following each
successful attempt).
6.1.4 Biometric functionality
In operational tests of previously deployed systems, the biometric functionality (i.e., enrolment, verification or
identification) under evaluation should be that of the deployed system. In operational tests of systems
deployed for the purpose of operational testing, the experimenter may determine which comparison
functionality(ies) to implement and evaluate. The rationale for selecting the comparison functionality
components to be evaluated by the operational test shall be reported.
NOTE An operational test can incorporate both identification and verification functions if, for example, data is used to
execute searches against watch lists and also for verification against an existing enrolment.
6.1.5 Performance measures
Performance measures relevant for operational evaluation include:
 throughput for enrolment and recognition transactions,
 failure-to-enrol rate,
 system rejection rate (in verification systems),
 system identification rate (in identification systems)
 false accept rate and false reject rate (in verification systems when the evaluation can establish ground
truth),
 false-positive identification error rate and false-negative identification error rate (in identification systems
when the evaluation can establish ground truth).
Considerations for determining and reporting these performance measures are specified in Clause 6.4. In
addition, experimenters shall determine any specific performance measures to be generated through the
operational test. Annex A provides a list of potential performance metrics.
6.2 Application characteristics
6.2.1 General
Application characteristics shall be considered in order to plan for test data collection that will be
representative of operational use. The following elements shall be addressed:
 concept of operations,
 guidance and instruction,
6 © ISO/IEC 2012 – All rights reserved

 levels of effort and decision policies,
 use of multiple instances,
 environment,
 deployment factors,
 acclimatization, and
 habituation.
6.2.2 Concept of operations
The concept of operations of the operational application being tested shall be determined and reported.
Description of the concept of operations shall include, but not be limited to, discussion of:
 integration of biometric systems with external systems such as logical or physical access systems;
 authentication methods and systems that the biometric system is replacing or complementing, if
applicable; and
 the category of fielded application under evaluation, e.g.:
 enrolment,
 physical access control,
 logical access control,
 surveillance or screening,
 identification, or
 examiner assisted identification.
6.2.3 Guidance and instruction
While operational test subjects should not be accompanied by test support personnel in their interaction with
the biometric system, as would be the case in a scenario evaluation, they may still be given instructions on
how to utilize the technology. Any instructions on how to utilize the biometric system shall be reported.
Information given to the test subject regarding the operational test shall also be reported.
The experimenter shall define the following elements related to guidance provided during enrolment and
recognition for each system tested:
 attended/unattended operation;
 information provided by attendant to test subject prior to interaction;
 type and extent of feedback provided by biometric system to test subject during interaction;
 type and extent of feedback provided by attendant to test subject during interaction;
 use of scripts, instructions, guidance tools, or other mechanisms to inform test subjects as to the optimal
method(s) of interacting with the system;
© ISO/IEC 2012 – All rights reserved 7

 correction and recording of improper interaction with device;
 constraints on test subject appearance and apparel; and
 differences in guidance and feedback from that provided for operational use.
NOTE For operational tests in which the system is not already in operational use prior to testing (i.e., a pilot
deployment), the amount of attendant-test subject interaction should approximate that which would be provided by the
biometric system vendor or administrative personnel for a typical deployment.
6.2.4 Levels of effort and decision policies
The level of effort involved in enrolment and recognition, as well as the test system’s decision policies, are
dictated by the operational environment. In most cases the system will have a specific configuration based on
the concept of operations and will execute a match/no match/enrol or other decision in real time, such that the
subject’s subsequent activities in an operational environment are contingent on the outcome.
For tests in which enrolment is in scope, the experimenter shall determine and report enrolment attempt limits
and decision policies for the system to the extent feasible. Such limits and policies may include the following:
 minimum and maximum number of attempts required or permitted to enrol (see Annex B). A system may
allow enrolment after one attempt, or may require multiple attempts.
 minimum and maximum duration permitted or required to enrol within a given enrolment attempt or
transaction. A system may terminate an enrolment transaction e.g., after a fixed duration. This may be
due to (1) the inability to capture or acquire a biometric sample of sufficient quality or distinction, or (2) the
inability to sense any biometric characteristic whatsoever. Incident (1) means that a biometric system has
acquired and processed data, but found it lacking; incident (2) means that the data was not acquired
and/or processed.
 whether the enrolment process includes an immediate attempt at verification using the enrolment to
confirm satisfactory enrolment and to help subject familiarization, and
 quality criteria or threshold applied during enrolment.
For tests in which recognition is in scope, the experimenter shall determine and report comparison attempt
limits and decision policies for system to the extent feasible. Such limits and policies may include the
following:
 minimum and maximum number of attempts required or permitted per comparison transaction. A system
may make a decision after one attempt, or may require multiple attempts.
 minimum and maximum duration permitted or required to match within a given comparison attempt or
transaction. A system may terminate a comparison transaction after a fixed duration. For systems that do
not time out, a time limit may be established for the purposes of testing.
 quality criteria and thresholds applied during comparison.
6.2.5 Multiple-instance systems
When it is known that a system utilizes multiple instances from the same test subject for the purposes of
enrolment or recognition this shall be reported. In multi-instance, multi-presentation, or multi-attempt systems,
fusion techniques may be used to combine information from two or more instances, presentations, and/or
attempts. The fusion techniques used, and the method in which they are applied, should be reported when
available.
8 © ISO/IEC 2012 – All rights reserved

NOTE In an operational system, an attempt in which the subject is rejected might be automatically followed by a
fallback attempt in which the system utilizes a different instance of the same modality from the same subject. For example,
a system might utilize the left index finger for comparison and prompt for placement of the right index finger if the left index
finger fails. This would typically apply to modalities such as fingerprint and iris in which most biometric capture subjects
can utilize more than one instance for enrolment and recognition.
6.2.6 Environment
Environmental conditions can play a major part in operational performance levels. All relevant environmental
conditions shall be considered. Reference should be made to ISO/IEC TR 19795-3 [5] which lists the physical
environmental conditions to be considered for different modalities and operational settings. These include, but
are not limited to:
 ambient temperature,
 atmospheric pressure,
 relative humidity,
 exposure to elements (e.g., rain, fog, snow, hail, etc.),
 illumination (including type, direction, intensity), and
 ambient (acoustic) noise.
Environmental conditions that are controlled by the operational system shall be reported to the extent feasible.
Any introduction of controls on environmental conditions not normally incorporated in the fielded application
should be avoided. If introduced, such controls shall be documented. Conditions that are not controlled may
be monitored and reported to the extent that these aspects are of interest.
EXAMPLE 1 If the operational system utilizes dedicated lighting for face recognition, details of the lighting system
would be reported.
EXAMPLE 2 If operational performance data is not collected on days where the temperature exceeds 30C, such
occurrences would be documented in the test report. Similarly, if a portable air conditioning unit were used on such days,
this would be reported.
For both monitoring and acceptance testing, it may be appropriate to record pertinent environmental data (e.g.,
ambient temperature, relative humidity, precipitation, cloudiness, illumination levels). The most comprehensive
and informative data may be collected at the individual location of the biometric device transaction (e.g., at
pedestrian gate N), and at the specific time of the test transaction. However, this approach might be complex,
difficult and expensive. An alternative approach may collect the environmental data at a representative
location (or set of locations) per test site, including a date-time stamp, and correlate the data off-line with the
individual transaction events.
6.2.7 Deployment factors
Deployment factors shall also be considered as these affect the usability of the system. Such factors can
include constraints for security, health and safety (e.g., having equipment and attendants on different sides of
a glass screen, or fastening equipment so it cannot be moved).
The experimenter should report the physical layout of the operational environment, including but not limited to
the following:
 dimensional area dedicated to test execution;
 positions of natural and artificial lighting;
 positioning of biometric acquisition devices.
© ISO/IEC 2012 – All rights reserved 9

6.2.8 Acclimatization
Acclimatization should mirror that of operational use. The manner and degree by which biometric
characteristics of the test crew are acclimatized to the operational environment shall be reported, and
justification shall be provided if this differs from acclimatization in operational use.
6.2.9 Habituation
The degree to which the subjects are habituated to device(s) under test prior to test execution shall be
determined and reported, where practical. Habituation may be reported in terms of the period of time over
which subjects have interacted with the specific device or class of devices; and the frequency of their use (e.g.,
several times daily, once daily, once weekly).
NOTE 1 Systems such as time and attendance and network login would in most cases have a habituated subject base.
Systems such as voter ID, border entry, and entitlements can require much less regular interaction with a biometric
system or device and therefore would be more likely to be categorized as non-habituated.
NOTE 2 Subject habituation to device(s) under test can have a substantial impact on error rates and throughput.
Habituated subjects will tend to generate lower false reject rates and failure-to-acquire rates, and shorter throughput times,
than non-habituated subjects.
Test crew habituation should be similar to that of the target population. A test crew can become habituated to
a device prior to testing, such as in the course of employment, or can become habituated to a device in the
course of testing.
Subject habituation is impacted by the frequency of subjects’ transactions as well as by operational
parameters of the system, such as threshold and maximum number of recognition attempts allowed per
transaction. The operational values of such parameters should therefore not be altered for testing.
For tests in which the biometric device is newly installed, experimenters should allow sufficient time for test
crew to become habituated before collecting data that is intended to predict the long-term performance of the
device. This time may be stipulated either as a specific period (for example, 2 weeks), the time required to
obtain a minimum average number of subject interactions, or by monitoring performance data to determine
when the test crew “learning curve” flattens out indicating that sufficient habituation has been obtained.
NOTE 3 If one aim of the test is to measure the effect of habituation on performance, more detailed controls and
reporting might be needed. See Annex F.
NOTE 4 If multiple devices (of the same or of different biometric modalities) are under test, test subjects might not be
equally habituated to all device types. For example, a test subject might be habituated to placement-based fingerprint
devices but not to swipe-based fingerprint devices. Further, certain biometric technologies are more heavily impacted by
the degree of habituation than others.
6.3 Test Plan
6.3.1 General
Operational test planning is directed by the type of performance information an organization wishes to collect
(addressed in Clause 6.1), and is constrained by characteristics of the application and the operational
environment (addressed in Clause 6.2) that should not be altered for the purpose of testing.
The test plan shall specify:
 system implementation and configuration;
 test population;
 test transactions from which performance metrics will be derived.
10 © ISO/IEC 2012 – All rights reserved

In determining the target number of test transactions to be made, it is important to consider how accurate the
results need to be and the level of confidence bounds to be used.
Annex C provides an outline test plan for an operational evaluation. Annex D provides guidance on statistical
methods for determining population sizes etc. Annex E describes elements of a test plan to monitor and
evaluate long-term and temporal trends in operational system performance. Annex F describes elements of a
test plan for operational habituation testing.
6.3.2 System implementation and configuration
Collection of data for the purposes of operational testing should be done in a manner that influences as little
as possible subject and public impressions of the system, and that has minimal impact on normal use and
operation.
The system under test may be configured or instrumented to produce data supporting performance analysis
(e.g., comparison scores, quality scores or sample images or features). The act of collecting such data will
often impact performance of the system itself, particularly with respect to throughput. The experimenter shall
(i) document any instrumentation and reconfiguration of the system; (ii) minimize the impact of these
modifications on performance; and (iii) estimate and document the impact on performance attributable to the
modifications.
EXAMPLE A system might be configured to log comparisons scores resulting from 1:1 comparisons, or to save
recognition samples. If saving samples noticeably increases transaction times above those of the operational system, this
could substantially affect subject interactions and hence error rates as well as throughput. If so, the saving of such data
would be inadvisable.
The operational application might constrain (i) what data can be recorded, and (ii) the degree to which the
operational system can be modified. For example, a physical access control system might be incapable of
storing samples or providing visibility into comparison functions due to architecture constraints. Conversely, a
border control application might mandate that samples and detailed transaction logs be retained. Moreover,
operational test outputs are generally recorded on a transactional basis, which can limit one’s ability to
evaluate performance at multiple comparison score thresholds.
NOTE The issue of visibility into device operations can be important to operational testing. Depending on the output
of the device, one might not know whether a rejected transaction was due to a biometric matching error, a time-out, a
failure-to-acquire or invalidity o
...