ISO/IEC TR 9126-4:2004

TECHNICAL ISO/IEC
REPORT TR
9126-4
First edition
2004-04-01
Software engineering — Product
quality —
Part 4:
Quality in use metrics
Génie du logiciel — Qualité des produits —
Partie 4: Qualité en métrologie d'usage

Reference number
©
ISO/IEC 2004
PDF disclaimer
This PDF file may contain embedded typefaces. In accordance with Adobe's licensing policy, this file may be printed or viewed but
shall not be edited unless the typefaces which are embedded are licensed to and installed on the computer performing the editing. In
downloading this file, parties accept therein the responsibility of not infringing Adobe's licensing policy. The ISO Central Secretariat
accepts no liability in this area.
Adobe is a trademark of Adobe Systems Incorporated.
Details of the software products used to create this PDF file can be found in the General Info relative to the file; the PDF-creation
parameters were optimized for printing. Every care has been taken to ensure that the file is suitable for use by ISO member bodies. In
the unlikely event that a problem relating to it is found, please inform the Central Secretariat at the address given below.

©  ISO/IEC 2004
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means,
electronic or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the address below or
ISO's member body in the country of the requester.
ISO copyright office
Case postale 56 • CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 749 09 47
E-mail copyright@iso.org
Web www.iso.org
Published in Switzerland
ii © ISO/IEC 2004 – All rights reserved

Contents
1 Scope . 1
2 Conformance. 2
3 Normative References. 2
4 Terms and definitions. 2
4.1 Context of use. 2
4.2 Goal. 2
4.3 Task . 3
5 Symbols and abbreviated terms . 3
6 Use of software quality metrics. 3
7 How to read and use the metrics tables. 4
8 Metrics Tables. 4
8.1 Effectiveness metrics. 6
8.2 Productivity metrics. 7
8.3 Safety metrics . 9
8.4 Satisfaction metrics. 11
Annex A (Informative) Considerations when using metrics . 12
A.1 Interpretation of measures. 12
A.2 Validation of metrics. 13
A.3 Use of metrics for estimation (judgement) and prediction (forecast). 15
A.4 Detecting deviations and anomalies in quality problem prone components. 16
A.5 Displaying measurement results. 16
Annex B (Informative) Use of Quality in Use, External & Internal Metrics (Framework Example). 17
B.1 Introduction . 17
B.2 Overview of development and quality process . 17
B.3 Quality Approach Steps . 18
Annex C (Informative) Detailed explanation of metric scale types and measurement types. 23
C.1 Metric scale types . 23
C.2 Measurement Types . 24
Annex D (Informative) Term(s) . 30
D.1 Definitions . 30
Annex E (Informative) Quality in use evaluation process . 32
E.1 Establish evaluation requirements. 32
E.2 Specify the evaluation. 33
E.3 Design the evaluation . 35
E.4 Execute the evaluation. 36
Annex F (Informative) Common Industry Format for Quality in Use Test Reports . 37
F.1 Purpose and Objectives. 37
F.2 Report Format Description. 38
F.3 References. 46
Annex G (Informative)  Common Industry Format Usability Test Example . 47
G.1 Introduction . 48
G.2 Method . 49
G.3 Results . 52
G.4 Appendix A – Participant Instructions. 58
© ISO/IEC 2004 – All rights reserved iii

Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are members of
ISO or IEC participate in the development of International Standards through technical committees
established by the respective organization to deal with particular fields of technical activity. ISO and IEC
technical committees collaborate in fields of mutual interest. Other international organizations, governmental
and non-governmental, in liaison with ISO and IEC, also take part in the work. In the field of information
technology, ISO and IEC have established a joint technical committee, ISO/IEC JTC 1.
International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2.
The main task of the joint technical committee is to prepare International Standards. Draft International
Standards adopted by the joint technical committee are circulated to national bodies for voting. Publication as
an International Standard requires approval by at least 75 % of the national bodies casting a vote.
In exceptional circumstances, the joint technical committee may propose the publication of a Technical Report
of one of the following types:
— type 1, when the required support cannot be obtained for the publication of an International Standard,
despite repeated efforts;
— type 2, when the subject is still under technical development or where for any other reason there is the
future but not immediate possibility of an agreement on an International Standard;
— type 3, when the joint technical committee has collected data of a different kind from that which is normally
published as an International Standard (“state of the art”, for example).
Technical Reports of types 1 and 2 are subject to review within three years of publication, to decide whether
they can be transformed into International Standards. Technical Reports of type 3 do not necessarily have to
be reviewed until the data they provide are considered to be no longer valid or useful.
Attention is drawn to the possibility that some of the elements of this document may be the subject of patent
rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights.
ISO/IEC TR 9126-4 which is a Technical Report of type 2, was prepared by Joint Technical Committee
ISO/IEC JTC 1, Information technology, Subcommittee SC 7, Software and system engineering.
ISO/IEC TR 9126 consists of the following parts, under the general title Software engineering — Product
quality :
 Part 1: Quality model
 Part 2: External metrics
 Part 3: Internal metrics
 Part 4: Quality in use metrics

iv © ISO/IEC 2004 – All rights reserved

Introduction
This Technical Report provides quality in use metrics for measuring attributes of quality in use defined in
ISO/IEC 9126-1. The metrics listed in this Technical Report are not intended to be an exhaustive set.
Developers, evaluators, quality managers and acquirers may select metrics from this Technical Report for
defining requirements, evaluating software products, measuring quality aspects and other purposes. They
may also modify the metrics or use metrics that are not included here. This report is applicable to any kind of
software product, although each of the metrics is not always applicable to every kind of software product.
ISO/IEC 9126-1 defines terms for the software quality characteristics and how these characteristics are
decomposed into subcharacteristics. ISO/IEC 9126-1, however, does not describe how any of these
subcharacteristics could be measured. ISO/IEC 9126-2 defines external metrics, ISO/IEC 9126-3 defines
internal metrics and ISO/IEC 9126-4 defines quality in use metrics, for measurement of the characteristics or
subcharacteristics. Internal metrics measure the software itself, external metrics measure the behaviour of the
computer-based system that includes the software, and quality in use metrics measure the effects of using the
software in a specific context of use.
This Technical Report is intended to be used together with ISO/IEC 9126-1. It is strongly recommended to
read ISO/IEC 14598-1 and ISO/IEC 9126-1, prior to using this Technical Report, particularly if the reader is
not familiar with the use of software metrics for product specification and evaluation.

© ISO/IEC 2004 – All rights reserved v

TECHNICAL REPORT ISO/IEC TR 9126-4:2004(E)

Software engineering — Product quality —
Part 4:
Quality in use metrics
1 Scope
This Technical Report defines quality in use metrics for the characteristics defined in ISO/IEC 9126-1, and is
intended to be used together with ISO/IEC 9126-1.
This Technical Report contains:
• an explanation of how to apply software quality metrics;
• a basic set of metrics for each characteristic;
• an example of how to apply metrics during the software product life cycle.
It includes as informative annexes a quality in use evaluation process and a reporting format.
This Technical Report does not assign ranges of values of these metrics to rated levels or to grades of
compliance, because these values are defined for each software product or a part of the software product, by
its nature, depending on such factors as category of the software, integrity level and users' needs. Some
attributes may have a desirable range of values, which does not depend on specific user needs but depends
on generic factors, i.e. human cognitive factors.
This Technical Report can be applied to any kind of software for any application. Users of this Technical
Report can select or modify and apply metrics and measures from this Technical Report or may define
application-specific metrics for their individual application domain. For example, the specific measurement of
quality characteristics such as safety or security may be found in International Standards or Technical Reports
provided by IEC 65 and ISO/IEC JTC1/SC 27.
Intended users of this Technical Report include:
• Acquirer (an individual or organization that acquires or procures a system, software product or software
service from a supplier);
• Evaluator (an individual or organization that performs an evaluation. An evaluator may, for example, be a
testing laboratory, the quality department of a software development organization, a government
organization or user);
• Developer (an individual or organization that performs development activities, including requirements
analysis, design and testing through acceptance during the software life cycle process);
• Maintainer (an individual or organization that performs maintenance activities);
• Supplier (an individual or organization that enters into a contract with the acquirer for the supply of a
system, software product or software service under the terms of the contract) when validating software
quality at qualification test;
• User (an individual or organization that uses the software product to perform a specific function) when
evaluating quality of software product at acceptance test;
© ISO/IEC 2004 – All rights reserved 1

• Quality manager (an individual or organization that performs a systematic examination of the software
product or software services) when evaluating software quality as part of quality assurance and quality
control.
2 Conformance
There are no conformance requirements in this Technical Report.
NOTE General conformance requirements for metrics are in ISO/IEC 9126-1.
3 Normative References
ISO 8402, Quality management and quality assurance — Vocabulary
ISO/IEC 9126, Software engineering — Product quality
ISO/IEC 9126-1, Software engineering — Product quality — Part 1: Quality model
ISO/IEC TR 9126-2, Software engineering — Product quality — Part 2: External metrics [Technical Report]
ISO/IEC TR 9126-3, Software engineering — Product quality — Part 3: Internal metrics [Technical Report]
ISO 9241-11:1998, Ergonomic requirements for office work with visual display terminals (VDTs) — Part 11:
Guidance on usability
ISO/IEC 14598-1, Information technology — Software product evaluation — Part 1: General overview
ISO/IEC 14598-3, Software engineering — Product evaluation — Part 3: Process for developers
ISO/IEC 14598-5:1998, Information technology — Software product evaluation — Part 5: Process for
evaluators
ISO/IEC 12207:1995, Information technology — Software life cycle processes
ISO/IEC 14143-1, Information technology — Software measurement — Functional size measurement —
Part 1: Definition of concepts
4 Terms and definitions
For the purposes of this Technical Report, the definitions contained in ISO/IEC 14598-1, ISO/IEC 9126-1 and
the following apply. Some of the definitions from ISO/IEC 14598-1 and ISO/IEC 9126-1 are reproduced in
Annex D.
4.1 context of use
the users, tasks, equipment (hardware, software and materials), and the physical and social environments in
which a product is used
[ISO 9241-11]
4.2 goal
an intended outcome
[ISO 9241-11]
2 © ISO/IEC 2004 – All rights reserved

4.3 task
the activities required to achieve a goal
NOTE 1 These activities can be physical or cognitive.
NOTE 2 Job responsibilities can determine goals and tasks.
[ISO 9241-11]
5 Symbols and abbreviated terms
The following symbols and abbreviated terms are used in this Technical Report:
• SQA - Software Quality Assurance (Group)
• SLCP – Software Life Cycle Processes
6 Use of software quality metrics
These Technical Reports (ISO/IEC 9126-2, ISO/IEC 9126-3 and ISO/IEC 9126-4) provide a suggested set of
quality metrics (external, internal and quality in use metrics) to be used with the ISO/IEC 9126-1 quality
model. The user of these reports may modify the metrics defined, and/or may also use metrics not listed.
When using a modified or a new metric not identified in these Technical Reports, the user should specify how
the metrics relate to the ISO/IEC 9126-1 quality model or any other substitute quality model that is being used.
The user of these Technical Reports should select the quality characteristics and subcharacteristics to be
evaluated from ISO/IEC 9126-1, identify the appropriate direct and indirect measures, identify the relevant
metrics and then interpret the measurement result in objective manner. The user of these Technical Reports
also may select product quality evaluation processes during the software life cycle from the ISO/IEC 14598
series of International Standards. These give methods for measurement, assessment and evaluation of
software product quality. They are intended for use by developers, acquirers and independent evaluators,
particularly those responsible for software product evaluation (see Figure 1).
effect of
software product
software product
influences
influences
internal
external quality in
quality
quality
use
contexts
depends on
depends on
of use
internal external quality in use
metrics metrics metrics
Figure 1 — Relationship between types of metrics
The internal metrics may be applied to a non-executable software product during its development stages
(such as request for proposal, requirements definition, design specification or source code). Internal metrics
provide the users with the ability to measure the quality of the intermediate deliverables and thereby predict
the quality of the final product. This allows the user to identify quality issues and take corrective action as
early as possible in the development life cycle.
The external metrics may be used to measure the quality of the software product by measuring the behaviour
of the system of which it is a part. The external metrics can only be used during the testing stages of the life
cycle process and during any operational stages. The measurement is performed when executing the
software product in the system environment in which it is intended to operate.
© ISO/IEC 2004 – All rights reserved 3

The quality in use metrics measure whether a product meets the needs of specified users to achieve specified
goals with effectiveness, productivity, safety and satisfaction in a specified context of use. This can be only
achieved in a realistic system environment.
User quality needs can be specified as quality requirements by quality in use metrics, by external metrics, and
sometimes by internal metrics. These requirements specified by metrics should be used as criteria when a
product is evaluated.
It is recommended to use internal metrics having a relationship as strong as possible with the target external
metrics, so that they can be used to predict the values of external metrics. However, it is often difficult to
design a rigorous theoretical model that provides a strong relationship between internal metrics and external
metrics. Therefore, a hypothetical model that may contain ambiguity may be designed and the extent of the
relationship may be modelled statistically during the use of metrics.
Recommendations and requirements related to validity and reliability are given in ISO/IEC 9126-1:A.4.
Additional detailed considerations when using metrics are given in Annex A of this Technical Report.
7 How to read and use the metrics tables
The metrics listed in clause 8 are categorised by the characteristics in ISO/IEC 9126-1. The following
information is given for each metric in the table:
a) Purpose of the metric: This is expressed as the question to be answered by the application of the metric.
b) Method of application: Provides an outline of the application.
c) Measurement, formula and data element computations: Provides the measurement formula and explains
the meanings of the used data elements.
NOTE In some situations more than one formula is proposed for a metric.
d) Interpretation of measured value: Provides the range and preferred values.
e) Metric scale type: Type of scale used by the metric. Scale types used are; Nominal scale, Ordinal scale,
Interval scale, Ratio scale and Absolute scale.
NOTE: A more detailed explanation is given in Annex C.
f) Measure type: Types used are; Size type (e.g. Function size, Source size) , Time type ( e.g. Elapsed time,
User time) , Count type ( e.g. Number of changes, Number of failures).
NOTE A more detailed explanation is given in Annex C.
g) Input to measurement: Source of data used in the measurement.
h) ISO/IEC 12207 SLCP Reference: Identifies software life cycle process(es) where the metric is applicable.
i) Target audience: Identifies the user(s) of the measurement results.
8 Metrics Tables
8.0 General
The metrics listed in this clause are not intended to be an exhaustive set and may not have been validated.
They are listed by software quality characteristic.
Metrics, which may be applicable, are not limited to these listed here. Additional specific metrics for particular
purposes are provided in other related documents, such as functional size measurement or precise time
efficiency measurement.
4 © ISO/IEC 2004 – All rights reserved

NOTE It is recommended to refer a specific metric or measurement from specific standards, Technical Reports or
guidelines Functional size measurement is defined in ISO/IEC 14143. An example of precise time efficiency measurement
can be referred from ISO/IEC 14756.
Metrics should be validated before application in a specific environment (see Annex A).
NOTE This list of metrics is not finalized, and may be revised in future versions of this Technical Report. Readers of
this Technical Report are invited to provide feedback.
The quality in use metrics in this clause measure the effectiveness, productivity, safety or satisfaction with
which specified users achieve specified goals in a specified context of use. Quality in use depends not only on
the software product, but also on the particular context in which the product is being used. The context of use
is determined by user factors, task factors and physical and social environmental factors.
Quality in use is assessed by observing representative users carrying out representative tasks in a realistic
context of use (see Annex E). The measures may be obtained by simulating a realistic working environment
(for instance in a usability laboratory) or by observing operational use of the product. In order to specify or
measure quality in use it is first necessary to identify each component of the intended context of use: the
users, their goals, and the environment of use. The evaluation should be designed to match this context of
use as closely as possible. It is also important that users are only given the type of help and assistance that
would be available to them in the operational environment.
NOTE The term usability is sometimes used with a similar meaning to quality in use (but excluding safety) (e.g. in
ISO 9241-11).
Some external usability metrics (ISO/IEC 9126-2) are tested in a similar way, but evaluate the use of
particular product features during more general use of the product to achieve a typical task as part of a test of
the quality in use.
Quality in use has four characteristics (effectiveness, productivity, safety and satisfaction) and no
subcharacteristics.
© ISO/IEC 2004 – All rights reserved 5

6 © ISO/IEC 2004 – All rights reserved
8.1 Effectiveness metrics
Effectiveness metrics assess whether the tasks performed by users achieve specified goals with accuracy and completeness in a specified context of use.
They do not take account of how the goals were achieved, only the extent to which they were achieved (see E.2.1.2).
Table 8.1 Effectiveness metrics
Metric Name Purpose of the Method of Measurement, formula and Interpretation of Metric Measure Input to 12207 Target
metrics application data element computations measured value scale type measurement reference audience
type
Task What proportion User test M1 = |1-ΣA | 0<= M1 <=1 — A= Operation 6.5 Validation User
i
effectiveness of the goals of proportion (test) report
A = proportional value of each missing or The closer to 1.0 5.3 Qualification
i
the task is  Human
the better. testing
incorrect component in the task output
achieved User interface
correctly? monitoring designer
5.4 Operation
record
NOTE Each potential missing or incomplete component is given a weight Ai based on the extent to which it detracts from the value of the output to the business or user. (If the sum of the
weights exceed 1, the metric is normally set to 0, although this may indicate negative outcomes and potential safety issues.) (See for example G.3.1.1.) The scoring scheme is refined iteratively by
applying it to a series of task outputs and adjusting the weights until the measures obtained are repeatable, reproducible and meaningful.
Task completion What proportion User test X = A/B 0<= X <=1 Ratio A = Count Operation 6.5 Validation User
of the tasks is B = Count (test) report
The closer to 5.3 Qualification
completed? A = number of tasks completed Human
1.0 the better. X = testing
B = total number of tasks attempted User interface
Count/Cou
monitoring designer
5.4 Operation
nt
record
NOTE This metric can be measured for one user or a group of users. If tasks can be partially completed the Task effectiveness metric should be used.
Error frequency What is the User test X = A/T 0<= X Absolute A = Count Operation 6.5 Validation User
frequency of  (test) report
The closer to 0 5.3 Qualification
errors? A = number of errors made by the user Human
the better. testing
User interface
T= time or number of tasks
monitoring designer
5.4 Operation
record
NOTE This metric is only appropriate for making comparisons if errors have equal importance, or are weighted.

© ISO/IEC 2004 – All rights reserved 7
8.2 Productivity metrics
Productivity metrics assess the resources that users consume in relation to the effectiveness achieved in a specified context of use. The most common
resource is time to complete the task, although other relevant resources could include the user’s effort, materials or the financial cost of usage.
Table 8.2 Productivity metrics
Metric Name Purpose of the Method of Measurement, formula and Interpretation of Metric Measure Input to 12207 Target
metrics application data element computations measured value scale type measurement reference audience
type
Task time How long does User test X = Ta 0<= X Interval T= Time Operation 6.5 Validation User
it take to (test) report
The smaller the 5.3 Qualification
complete a Ta = task time Human
better. testing
task? User interface
monitoring designer
5.4 Operation
record
Task efficiency How efficient User test X = M1 / T 0<= X — T= Time Operation 6.5 Validation User
are the users?  X= (test) report
5.3 Qualification
M1 = task effectiveness The larger the proportion/ Human
testing
T = task time better. time User interface
monitoring designer
5.4 Operation
record
NOTE 1 Task efficiency measures the proportion of the goal achieved for every unit of time. Efficiency increases with increasing effectiveness and reducing task time. It enables comparisons to
be made, for example between fast error-prone interfaces and slow easy interfaces (see for example F.2.4.4).
NOTE 2 If Task completion has been measured, task efficiency can be measured as Task completion/task time. This measures the proportion of users who were successful for every unit of time.
A high value indicates a high proportion of successful users in a small amount of time.
Economic How cost- User test X = M1 / C 0<= X — C=value Operation 6.5 Validation User
productivity effective is the (test) report
The larger the X= 5.3 Qualification
user? M1 = task effectiveness Human
better. proportion/ testing
C = total cost of the task User interface
value
monitoring designer
5.4 Operation
record
NOTE Costs could for example include the user’s time, the time of others giving assistance, and the cost of computing resources, telephone calls, and materials

8 © ISO/IEC 2004 – All rights reserved

Metric Name Purpose of the Method of Measurement, formula and Interpretation of Metric Measure Input to 12207 Target
metrics application data element computations measured value scale type measurement reference audience
type
Productive What proportion User test X = Ta / Tb 0<= X <=1 Absolute Ta=Time Operation 6.5 Validation User
proportion of the time is the Tb=Time (test) report
The closer to 5.3 Qualification
user performing X= Time/ Human
Ta = productive time = 1.0 the better. testing
productive Time User interface
task time - help time - error time - search
actions? monitoring designer
5.4 Operation
time
record
Tb = task time
NOTE This metric requires detailed analysis of a videotape of the interaction (see Macleod M, Bowden R, Bevan N and Curson I (1997) The MUSiC Performance Measurement method,
Behaviour and Information Technology, 16, 279-293.)
Relative user How efficient is User test Relative user efficiency X = A / B 0<= X <=1 Absolute X = Operation 6.5 Validation User
efficiency a user proportion/ (test) report
The closer to 5.3 Qualification
compared to an proportion Human
A = ordinary user’s task efficiency 1.0 the better. testing
expert? User interface
B = expert user’s task efficiency
monitoring designer
5.4 Operation
record
NOTE The user and expert carry out the same task. If the expert was 100 % productive, and the user and expert had the same task effectiveness, this metric would give a
similar value to the Productive proportion.

© ISO/IEC 2004 – All rights reserved 9
8.3 Safety metrics
Safety metrics assess the level of risk of harm to people, business, software, property or the environment in a specified context of use. It includes the health
and safety of the both the user and those affected by use, as well as unintended physical or economic consequences.

Table 8.3 Safety metrics
Metric Name Purpose of the Method of Measurement, formula and Interpretation of Metric Measure Input to 12207 Target
metrics application data element computations measured value scale type type measurement reference audience
User health and What is the Usage X = 1-A / B 0<= X <=1 Absolute A = count Usage 5.4 Operation User
safety incidence of statistics  B = count monitoring
The closer to 1
health problems X = count/ record Human
A = number of users reporting RSI the better.
among users of count interface
B = total number of users
the product? designer
NOTE Health problems can include Repetitive Strain Injury, fatigue, headaches, etc.
Safety of people What is the Usage X = 1-A / B 0<= X <=1 Absolute A = count Usage 5.3 Qualification User
affected by use incidence of statistics  B = count monitoring testing
The closer to 1
of the system hazard to X = count/ record Human
A = number of people put at hazard the better. 5.4 Operation
people affected count interface
B = total number of people potentially
by use of the designer
affected by the system
system?
Developer
NOTE An example of this metric is Patient Safety, where A = number of patients with incorrectly prescribed treatment and B = total number of patients
Economic What is the Usage X = 1-A/ B 0<= X <=1 Absolute A = count Usage 5.4 Operation User
damage incidence of statistics B = count monitoring
The closer to 1
economic X = count/ record Human
A = number of occurrences of economic the better.
damage? count interface
damage
designer
B = total number of usage situations

Developer
NOTE This can also be measured based on the number of occurrences of situations where there was a risk of economic damage

10 © ISO/IEC 2004 – All rights reserved

Metric Name Purpose of the Method of Measurement, formula and Interpretation of Metric Measure Input to 12207 Target
metrics application data element computations measured value scale type measurement reference audience
type
Software damage What is the Usage X = 1-A / B 0<= X <=1 Absolute A = count Usage 5.4 Operation User
incidence of statistics  B = count monitoring
The closer to 1
software X = count/ record Human
A = number of occurrences of software the better.
corruption? count interface
corruption
designer
B = total number of usage situations

Developer
NOTE 1 This can also be measured based on the number of occurrences of situations where there was a risk of software damage
NOTE 2 Can also be measured as X = cumulative cost of software corruption/usage time.

© ISO/IEC 2004 – All rights reserved 11
8.4 Satisfaction metrics
Satisfaction metrics assess the user’s attitudes towards the use of the product in a specified context of use.
NOTE: Satisfaction is influenced by the user's perception of properties of the software product (such as those measured by external metrics) and by the user's perception of
the efficiency, productivity and safety in use.
Table 8.4 Satisfaction metrics
Metric Name Purpose of the Method of Measurement, formula and Interpretation of Metric Measure type Input to 12207 Target
metrics application data element computations measured value scale measurement reference audience
type
Satisfaction How satisfied is User test X = A/B 0 scale the user? A = questionnaire producing the larger the X= Count (test) report
5.3 Qualification
psychometric scales better Human
testing
User interface
B = population average
monitoring designer
5.4 Operation
record
Developer
NOTE Examples of psychometric questionnaires can be found in F.3.
Satisfaction How satisfied is User test X = ∑(A )/n Compare with Ord. A= Count Operation 6.5 Validation User
i
questionnaire the user with A = response to a question previous X= Count (test) report
i
5.3 Qualification
specific n = number of responses values, or with Human
testing
software population User interface
features? average monitoring designer
5.4 Operation
record
Developer
NOTE If the questionnaire items are combined to give an overall score, they should be weighted, as different questions may have different importance.
Discretionary What proportion Observation of X = A/B 0<=X<=1 Ratio A = Count Operation 6.5 Validation User
usage of potential usage The closer to 1 B = Count (test) report
A= number of times that specific software 5.3 Qualification
users choose to the better Human
functions/applications/systems are used X = User testing
use the interface
Count/Count monitoring
system? designer
B = number of times they are intended to 5.4 Operation
record
be used
NOTE This metric is appropriate when usage is discretionary.

Annex A
(Informative)
Considerations when using metrics
A.1 Interpretation of measures
A.1.1 Potential differences between test and operational contexts of use
When planning the use of metrics or interpreting measures it is important to have a clear understanding of the
intended context of use of the software, and any potential differences between the test and operational
contexts of use. For example, the “time required to learn operation" measure is often different between skilled
operators and unskilled operators in similar software systems. Examples of potential differences are given
below.
a) Differences between testing environment and the operational environment
Are there any significant differences between the testing environment and the operational environment?
The following are examples:
• testing with higher/comparable/lower performance of CPU of operational computer;
• testing with higher/comparable/lower performance of operational network and communication;
• testing with higher/comparable/lower performance of operational operating system;
• testing with higher/comparable/lower performance of operational user interface.
b) Differences between testing execution and actual operational execution
Are there any significant differences between the testing execution and operational execution in user
environment?
The following are examples:
• coverage of functionality in test environment;
• test case sampling ratio;
• automated testing of real time transactions;
• stress loads;
• 24 hour 7 days a week (non stop) operation;
• appropriateness of data for testing of exceptions and errors;
• periodical processing;
• resource utilisation;
• levels of interruption;
• production pressures;
• distractions.
12 © ISO/IEC 2004 – All rights reserved

c) User profile under observation
Are there any significant differences between test user profiles and operational user profiles?
The following are examples of types of users;
• user skill levels;
• specialist users or average users;
• limited user group or public users.
A.1.2 Issues affecting validity of results
The following issues may affect the validity of the data that is collected:
a) procedures for collecting evaluation results:
• automatically with tools or facilities/ manually collected/questionnaires or interviews;
b) source of evaluation results:
• developers' self reports/reviewers’ report/evaluator’s report;
c) results data validation:
• developers' self check/inspection by independent evaluators.
A.1.3 Balance of measurement resources
Is the balance of measures used at each stage appropriate for the evaluation purpose?
It is important to balance the effort used to apply an appropriate range of metrics for internal, external and
quality in use measures.
A.1.4 Correctness of specification
Are there significant differences between the software specification and the real operational needs?
Measurements taken during software product evaluation at different stages are compared against product
specifications. Therefore, it is very important to ensure by verification and validation that the product
specifications used for evaluation reflect the actual and real needs in operation A.2.
A.2 Validation of metrics
A.2.1 Desirable properties for metrics
To obtain valid results from a quality evaluation, the metrics should have the properties stated below. If a
metric does not have these properties, the metric description should explain the associated constraint on its
validity and, as far as possible, how that situation can be handled.
a) Reliability (of metric): Reliability is associated with random error. A metric is free of random error if
random variations do not affect the results of the metric
b) Repeatability (of metric): repeated use of the metric for the same product using the same evaluation
specification (including the same environment), type of users, and environment by the same evaluators,
should produce the same results within appropriate tolerances. The appropriate tolerances should include
such things as fatigue, and learning effect
© ISO/IEC 2004 – All rights reserved 13

c) Reproducibility (of metric): use of the metric for the same product using the same evaluation
specification (including the same environment) type of users and environment by different evaluators,
should produce the same results within appropriate tolerances.
NOTE It is recommended to use statistical analysis to measure the variability of the results.
d) Availability (of metric): The metric should clearly indicate the conditions (e.g. presence of specific
attributes) which constrain its usage.
e) Indicativeness (of metric): Capability of the metric to identify parts or items of the software which should
be improved, given the measured results compared to the expected ones.
NOTE The selected or proposed metric should provide documented evidence of the availability of the metric for use,
unlike those requiring project inspection only.
f) Correctness (of measure): The metric should have the following properties:
1) Objectivity (of measure): the metric results and its data input should be factual: i.e., not influenced
by the feelings or the opinions of the evaluator, test users, etc. (except for satisfaction or
attractiveness metrics where user feelings and opinions are being measured).
2) Impartiality (of measure): the measurement should not be biased towards any particular result.
3) Sufficient precision (of measure): Precision is determined by the design of the metric, and
particularly by the choice of the material definition used as the basis for the metric. The metric user
will describe the precision and the sensitivity of the metric.
g) Meaningfulness (of measure): the measurement should produce meaningful results about the software
behaviour or quality characteristics.
The metric should also be cost effective: that is, more costly metrics should provide higher value results.
A.2.2 Demonstrating the Validity of Metrics
The users of metrics should identify the methods for demonstrating the validity of metrics, as shown below.
a) Correlation
The variation in the quality characteristics values (the measures of principal metrics in operation
...