ISO/IEC 25022:2016

INTERNATIONAL ISO/IEC
STANDARD 25022
First edition
2016-06-15
Systems and software engineering —
Systems and software quality
requirements and evaluation (SQuaRE)
— Measurement of quality in use
Ingénierie des systèmes et du logiciel — Exigences de qualité et
évaluation des systèmes et du logiciel (SQuaRE) — Mesurage de la
qualité lors de l’utilisation
Reference number
©
ISO/IEC 2016
© ISO/IEC 2016, Published in Switzerland
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized otherwise in any form
or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior
written permission. Permission can be requested from either ISO at the address below or ISO’s member body in the country of
the requester.
ISO copyright office
Ch. de Blandonnet 8 • CP 401
CH-1214 Vernier, Geneva, Switzerland
Tel. +41 22 749 01 11
Fax +41 22 749 09 47
copyright@iso.org
www.iso.org
ii © ISO/IEC 2016 – All rights reserved

Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Conformance . 2
3 Normative references . 2
4 Terms and definitions . 2
5 Abbreviated terms . 6
6 Use of quality in use measures. 6
6.1 Applications of quality in use . 6
6.2 Measurement of quality in use . 7
6.3 Interpretation of quality in use measures . 8
6.4 Selecting appropriate quality in use measures . 9
6.5 Aspects of quality in use described in other International Standards . 9
7 Format used for documenting the quality measures . 9
8 Quality in use measures .10
8.1 General .10
8.2 Effectiveness measures .10
8.3 Efficiency measures .12
8.4 Satisfaction measures .13
8.4.1 General.13
8.4.2 Usefulness measures .14
8.4.3 Trust measures .15
8.4.4 (User experience) pleasure measures .15
8.4.5 (Ergonomic) comfort measures .16
8.5 Freedom from risk measures .16
8.5.1 General.16
8.5.2 Economic risk mitigation measures .18
8.5.3 Health and safety risk mitigation measures .19
8.5.4 Environmental risk mitigation measures .20
8.6 Context coverage measures .21
8.6.1 General.21
8.6.2 Context completeness measures .21
8.6.3 Flexibility measures .21
Annex A (informative) Examples of how to measure context coverage.23
Annex B (informative) Normalization of quality in use measures .25
Annex C (informative) Use of ISO/IEC 25022 for measuring usability in ISO 9241‑11 .30
Annex D (informative) Quality in use evaluation process .31
Annex E (informative) Relationship between different quality models .37
Annex F (informative) Quality measurement concepts .38
Annex G (informative) QMEs used to define quality measures .39
Bibliography .40
© ISO/IEC 2016 – All rights reserved iii

Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical
activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international
organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the
work. In the field of information technology, ISO and IEC have established a joint technical committee,
ISO/IEC JTC 1.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for
the different types of document should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent
rights. Details of any patent rights identified during the development of the document will be in the
Introduction and/or on the ISO list of patent declarations received (see www.iso.org/patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation on the meaning of ISO specific terms and expressions related to conformity
assessment, as well as information about ISO’s adherence to the WTO principles in the Technical
Barriers to Trade (TBT) see the following URL: Foreword - Supplementary information
The committee responsible for this document is ISO/IEC JTC 1, Information technology, SC 7, Software
and systems engineering.
This first edition of ISO/IEC 25022, which is a part of the SQuaRE series of standards, cancels and
replaces ISO/IEC 9126-4:2004, with the following changes:
— measures are given for the revised quality model for quality in use in ISO/IEC 25010;
— measures are categorized as generally applicable, could be used in a wide range of situations, or
specialized for specific needs;
— annexes that were common to ISO/IEC 9126-2, ISO/IEC 9126-3, and ISO/IEC 9126-4 have been
removed (and might be included in a future revision of ISO/IEC 25020).
The SQuaRE series of standards consists of the following divisions under the general title Systems and
software quality requirements and evaluation:
— ISO/IEC 2500n — Quality Management Division
— ISO/IEC 2501n — Quality Model Division
— ISO/IEC 2502n — Quality Measurement Division
— ISO/IEC 2503n — Quality Requirements Division
— ISO/IEC 2504n — Quality Evaluation Division
— ISO/IEC 25050 — 25099 SQuaRE Extension Division
Annexes A, B, C, D, E, F and G are for information only.
iv © ISO/IEC 2016 – All rights reserved

Introduction
This International Standard is a part of the SQuaRE series of International Standards. It provides a
set of measures for the characteristics of quality in use (defined in ISO/IEC 25010) that can be used
for specifying quality in use requirements (in conjunction with ISO/IEC 25030) and measuring and
evaluating quality in use (in conjunction with ISO/IEC 25040 and ISO/IEC 25041).
The quality measures included in this International Standard were selected based on their practical
value. They are based on established practice (including, for example, Reference [17]). They are not
intended to be exhaustive, and users of this International Standard are encouraged to refine them, if
necessary.
Quality Measurement Division
This International Standard is a part of ISO/IEC 2502n Quality Measurement Division of SQuaRE series
that currently consists of the following International Standards:
— ISO/IEC 25020 — Measurement reference model and guide: provides a reference model and guide
for measuring the quality characteristics defined in ISO/IEC 2501n Quality Model Division;
— ISO/IEC 25021 — Quality measure elements: provides a format for specifying Quality Measure
Elements and some examples of QMEs that can be used to construct software quality measures;
— ISO/IEC 25022 — Measurement of quality in use: provides measures, including associated
measurement functions for the quality characteristics in the quality in use model;
— ISO/IEC 25023 — Measurement of system and software product quality: provides measures,
including associated measurement functions and QMEs for the quality characteristics in the product
quality model;
— ISO/IEC 25024 — Measurement of data quality: provides measures, including associated
measurement functions and QMEs for the quality characteristics in the data quality model.
Figure 1 depicts the relationship between this International Standard and the other standards in the
ISO/IEC 2502n division.
© ISO/IEC 2016 – All rights reserved v

ISO/IEC 25020:
Measurement reference
model and guide
25020 gives guide for developing and
specifying Quality Measures
ISO/IEC 25022:         ISO/IEC 25023:
Measurement of system
Measurement of        ISO/IEC 25024:
ISO/IEC 25025:
and software product quality
quality in use
Measurement of Measurement of
-Revision
-Revision of 9126-4
data quality IS Service quality
of ISO/IEC 9126-2 & 9126-3
ISO/IEC 9126-4
ISO/IEC 25021 is used for developing and
specifying
ISO/IEC 25021:
Quality measure
Revision of
Figure 1 — Structure of the Quality Measurement Division
Outline and Organization of SQuaRE Series
The SQuaRE series consists of five main divisions and extension division. The outline of each divisions
within the SQuaRE series are as follows.
— ISO/IEC 2500n — Quality Management Division. The standards that form this division define
all common models, terms, and definitions referred further by all other standards from SQuaRE
series. The division also provides requirements and guidance for the planning and management of
a project.
— ISO/IEC 2501n — Quality Model Division. The standards that form this division provide quality
models for system/software products, quality in use, and data. An IT service quality model is under
development. Practical guidance on the use of the quality model is also provided.
— ISO/IEC 2502n — Quality Measurement Division. The standards that form this division include a
system/software product quality measurement reference model, definitions of quality measures,
and practical guidance for their application. This division presents internal measures of software
quality, external measures of software quality, and quality in use measures. Quality measure
elements forming foundations for the quality measures are defined and presented.
— ISO/IEC 2503n — Quality Requirements Division. The standard that forms this division helps
specifying quality requirements. These quality requirements can be used in the process of quality
requirements elicitation for a system/software product to be developed, designing a process for
achieving necessary quality, or as inputs for an evaluation process.
— ISO/IEC 2504n — Quality Evaluation Division. The standards that form this division provide
requirements, recommendations, and guidelines for system/software product evaluation, whether
performed by independent evaluators, acquirers, or developers. The support for documenting a
measure as an Evaluation Module is also presented.
ISO/IEC 25050 to ISO/IEC 25099 are reserved for SQuaRE extension International Standards, which
currently include ISO/IEC 25051 and the ISO/IEC 25060 to ISO/IEC 25069.
vi © ISO/IEC 2016 – All rights reserved

INTERNATIONAL STANDARD ISO/IEC 25022:2016(E)
Systems and software engineering — Systems and
software quality requirements and evaluation (SQuaRE) —
Measurement of quality in use
1 Scope
This International Standard defines quality in use measures for the characteristics defined in
ISO/IEC 25010, and is intended to be used together with ISO/IEC 25010. It can be used in conjunction
with the ISO/IEC 2503n and the ISO/IEC 2504n standards or to more generally meet user needs with
regard to product or system quality.
This International Standard contains the following:
— a basic set of measures for each quality in use characteristic;
— an explanation of how quality in use is measured.
This International Standard provides a suggested set of quality in use measures to be used with the
quality in use model in ISO/IEC 25010. They are not intended to be an exhaustive set.
It includes as informative annexes examples of how to measure context coverage (Annex A), options
for normalising quality in use measures (Annex B), use of ISO/IEC 25022 for measuring usability in
ISO 9241-11 (Annex C), a quality in use evaluation process (Annex D), the relationship between different
quality models (Annex E), and quality measurement concepts (Annex F).
The measures are applicable to the use of any human-computer system, including both computer
systems in use and software products that form part of the system.
This International Standard does not assign ranges of values of the measures to rated levels or to grades
of compliance because these values are defined for each system or product depending, on the context of
use and users’ needs.
Some attributes could have a desirable range of values, which does not depend on specific user needs
but depends on generic factors, for example, human cognitive factors.
The proposed quality in use measures are primarily intended to be used for quality assurance and
management of systems and software products based on their effects when actually used. The main
users of the measurement results are people managing development, acquisition, evaluation, or
maintenance of software and systems.
The main users of this International Standard are people carrying out specification and evaluation
activities as part of the following:
— development: including requirements analysis, design, and testing through acceptance during the
life cycle process;
— quality management: systematic examination of the product or computer system, for example, when
evaluating quality in use as part of quality assurance and quality control;
— supply: a contract with the acquirer for the supply of a system, software product, or software service
under the terms of a contract, for example, when validating quality at qualification test;
— acquisition: including product selection and acceptance testing, when acquiring or procuring a
system, software product, or software service from a supplier;
— maintenance: improvement of the product based on quality in use measures.
© ISO/IEC 2016 – All rights reserved 1

2 Conformance
Any quality requirement specification or quality evaluation that conforms to this International
Standard shall:
a) select the quality in use characteristics and/or subcharacteristics to be specified or evaluated as
defined in ISO/IEC 25010;
b) for each selected characteristic or subcharacteristic, all the General (G) quality measures defined
in Clause 8 should be used. If any are excluded, then provide a rationale;
c) select any Special (S) quality measures that are relevant;
d) if any quality measure is modified, provide the rationale for any changes;
e) define any additional quality measures to be used that are not included in this International
Standard;
f) define precisely how each quality measure is operationalized (for example details of the
measurement method or questionnaire used).
NOTE It is important to use the same measurement method when making comparisons.
3 Normative references
The following documents, in whole or in part, are normatively referenced in this document and are
indispensable for its application. For dated references, only the edition cited applies. For undated
references, the latest edition of the referenced document (including any amendments) applies.
ISO/IEC 25010, Systems and software engineering — Systems and software Quality Requirements and
Evaluation (SQuaRE) — System and software quality models
4 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO/IEC 25000 and ISO/IEC 25010
and the following apply.
NOTE The essential definitions from ISO/IEC 25000 SQuaRE series and the other ISO standards are
reproduced here.
4.1
context completeness
degree to which a product or system can be used with the required levels of effectiveness, efficiency,
satisfaction, and freedom from risk in each of the specified contexts of use
Note 1 to entry: Context completeness is a subcharacteristic of context coverage.
[SOURCE: ISO/IEC 25010:2011, modified — Added “the required levels of” and changed “all” to “each of”
for clarification.]
4.2
context coverage
degree to which a product or system can be used with effectiveness, efficiency, satisfaction, and freedom
from risk in both specified contexts of use and in contexts beyond those initially explicitly identified
Note 1 to entry: Context of use is relevant to both quality in use and some product quality (sub)characteristics
(where it is referred to as “specified conditions”).
[SOURCE: ISO/IEC 25010:2011, 4.1.5]
2 © ISO/IEC 2016 – All rights reserved

4.3
customer
relationship with the supplier of an organization or person that receives or uses a product or service
Note 1 to entry: The relationship can include a warranty or agreeing to the terms and conditions of a service.
4.4
effectiveness
accuracy and completeness with which users achieve specified goals
[SOURCE: ISO 9241-11:1998]
4.5
efficiency
resources expended in relation to the accuracy and completeness with which users achieve goals
Note 1 to entry: Relevant resources can include time to complete the task (human resources), materials, or the
financial cost of usage.
[SOURCE: ISO 9241-11:1998, modified — note 1 to entry added]
4.6
context of use
users, tasks, equipment (hardware, software and materials), and the physical and social environments
in which a system, product or service is used
[SOURCE: ISO 9241-11:1998, 3.5, modified — With “product” replaced by “system, product or service”.]
4.7
flexibility
degree to which a product or system can be used with acceptable levels of effectiveness, efficiency,
freedom from risk, and satisfaction in contexts beyond those initially specified in the requirements
Note 1 to entry: Flexibility is a subcharacteristic of context coverage.
[SOURCE: ISO/IEC 25010:2011, modified — Added “acceptable levels of” for clarification.]
4.8
formative evaluation
evaluation designed and used to improve the object of evaluation, especially when it is still being
developed
[SOURCE: ISO/TS 18152:2010, 4.6]
4.9
freedom from risk
degree to which the quality of a product or system mitigates or avoids potential risks to economic
status, human life, health, or the environment
Note 1 to entry: Risk is a function of the probability of occurrence of a given threat and the potential adverse
consequences of that threat’s occurrence.
Note 2 to entry: The risks considered by the SQuaRE series are those arising from insufficient product quality.
Note 3 to entry: Freedom from risk includes reduction of potential risks to the user, organisation or project.
[SOURCE: ISO/IEC 25010:2011, modified — Added “quality of” and “or avoids” for clarification.]
4.10
goal
intended outcome
[SOURCE: ISO 9241-11:1998]
© ISO/IEC 2016 – All rights reserved 3

4.11
measure (noun)
variable to which a value is assigned as the result of measurement
Note 1 to entry: The term “measures” is used to refer collectively to base measures, derived measures, and
indicators.
Note 2 to entry: In this International Standard, when the word “measure” is used qualified by a characteristic or
subcharacteristic, it refers to a quality measure.
[SOURCE: ISO/IEC 15939:2007, modified — note 2 to entry added]
4.12
measurement
set of operations having the object of determining a value of a measure
Note 1 to entry: Measurement can include assigning a qualitative category such as the language of a source
program (ADA, C, COBOL, etc.).
[SOURCE: ISO/IEC 15939:2007, modified — note 1 to entry modified from original]
4.13
measurement function
algorithm or calculation performed to combine two or more quality measurement elements
[SOURCE: ISO/IEC 25021:2012]
4.14
psychometrics
field of study concerned with the theory and technique for developing valid and reliable psychological
measures
4.15
quality in use
degree to which a product or system can be used by specific users to meet their needs to achieve specific
goals with effectiveness, efficiency, satisfaction, and freedom from risk in specific contexts of use
Note 1 to entry: The quality in use of a software product or system can be measured and evaluated by the effect
of the target system or software products when used by users of the implemented system or during field testing
or prototype testing.
Note 2 to entry: When quality in use is specified, it relates to specified users meeting their needs to achieve
specified goals with effectiveness, efficiency, satisfaction, and freedom from risk in specified contexts of use.
[SOURCE: ISO/IEC 25010:2011, modified — notes 1 and 2 to entry added]
4.16
quality measure
measure that is defined as a measurement function of two or more values of quality measure elements
[SOURCE: ISO/IEC 25021:2012]
4.17
quality measure element
measure defined in terms of a property and the measurement method for quantifying it, including
optionally the transformation by a mathematical function
[SOURCE: ISO/IEC 25021:2012]
4 © ISO/IEC 2016 – All rights reserved

4.18
quality model
defined set of characteristics, and of relationships between them, which provides a framework for
specifying quality requirements and evaluating quality
[SOURCE: ISO/IEC 25000:2014]
4.19
satisfaction
degree to which user needs are satisfied when a product or system is used in a specified context of use
Note 1 to entry: For a user who does not directly interact with the product or system, only purpose accomplishment
and trust are relevant.
Note 2 to entry: Satisfaction is the user’s response to interaction with the product or system, and includes
attitudes towards use of the product.
Note 3 to entry: Users include: primary users who interact with the system to achieve the primary goals,
secondary users who provide support, and indirect users who receive output, but do not interact with the system.
Note 4 to entry: In this International Standard, user’s needs include their desires and expectations associated
with use of a product, system, or service. Exceeding desires and expectations is a means of significantly
increasing satisfaction and improving the user experience.
[SOURCE: ISO/IEC 25010:2011, modified — notes 3 and 4 to entry added]
4.20
stakeholder satisfaction
degree to which stakeholder needs are satisfied when a product or system is used in a specified
context of use
Note 1 to entry: Users of a product or system are one type of stakeholder, so user satisfaction is one type of
stakeholder satisfaction.
[SOURCE: ISO/IEC 25010:2011, modified — Definition for the term “satisfaction” modified to refer to
stakeholders.]
4.21
summative evaluation
evaluation designed to present conclusions about the merit or worth of the object of evaluation
Note 1 to entry: The results can be used to produce recommendations about whether it should be retained,
altered, or eliminated.
Note 2 to entry: It is possible to design a method to provide a combined formative and summative evaluation.
Note 3 to entry: A summative test method is used to perform a summative evaluation.
[SOURCE: ISO/TS 20282-2:2013, 4.17]
4.22
system
combination of interacting elements organised to achieve one or more stated purposes
Note 1 to entry: A system may be considered as a product or as the services it provides.
Note 2 to entry: In practice, the interpretation of its meaning is frequently clarified by the use of an associative
noun, e.g. aircraft system. Alternatively, the word system may be substituted simply by a context dependent
synonym, e.g. aircraft, though this may then obscure a system principles perspective.
[SOURCE: ISO/IEC 15288:2015, modified — note 3 to entry deleted.]
© ISO/IEC 2016 – All rights reserved 5

4.23
task
activities required to achieve a goal
Note 1 to entry: These activities can be physical or cognitive.
Note 2 to entry: Job responsibilities can determine goals and tasks.
[SOURCE: ISO 9241-11:1998]
4.24
usability
degree to which a product or system can be used by specified users to achieve specified goals with
effectiveness, efficiency, and satisfaction in a specified context of use
Note 1 to entry: Adapted from ISO 9241-210.
Note 2 to entry: Usability can either be specified or measured as a product quality characteristic in terms of its
subcharacteristics, or specified or measured directly by measures that are a subset of quality in use.
[SOURCE: ISO/IEC 25010:2011]
4.25
use error
act or omission of an act that results in a different system response than intended by the manufacturer
or expected by the user
[SOURCE: IEC 62366:2007, modified — With “medical device” replaced by “system”.]
4.26
user
individual or group that benefits from a system during its utilization
[SOURCE: ISO/IEC 15939:2007]
5 Abbreviated terms
The following abbreviation is used in this International Standard.
QME Quality Measure Element
6 Use of quality in use measures
6.1 Applications of quality in use
This International Standard provides suggested measures for the characteristics and subcharacteristics
in the ISO/IEC 25010 quality in use model.
— Effectiveness
— Efficiency
— Satisfaction
— Usefulness
— Trust
— Pleasure (user experience)
6 © ISO/IEC 2016 – All rights reserved

— Comfort (ergonomic)
— Freedom from risk
— Economic risk mitigation
— Health and safety risk mitigation
— Environmental risk mitigation
— Context coverage
— Context completeness
— Flexibility
Quality in use is the degree to which a product or system can be used by specific users to meet their
needs to achieve specific goals with effectiveness, efficiency, and satisfaction and freedom from risk in
specific contexts of use.
In the quality in use model, the term usability refers to the subset of quality in use composed of
effectiveness, efficiency, satisfaction, and context coverage. Measures of the usability and functional
suitability of the user interface and interaction are contained in ISO/IEC 25023.
Unacceptable levels of freedom from risk can result from poor levels of usability, which can be caused
by poor levels of product usability. Unacceptable levels of freedom from risk can also be caused by poor
levels of other product quality characteristics.
Measures of quality in use measure the outcomes of interaction between a user and a system. The final
quality in use of an implemented system or software product can be measured when the system or
software product is used in the user’s environment for its intended purpose. Quality in use measures
can also be used at conceptual design and development stages as follows.
— Requirements: quality in use requirements can be specified early in the design and development
process, to give a high level goal for the intended quality to be experienced by users and other
stakeholders.
— Formative evaluation of prototypes: specific problems with quality in use can be identified by
testing prototypes with small numbers of representative users during development using product
usability measures and unnormalized quality in use measures in order to identify problems and
improve the product.
— Summative evaluation of prototypes: estimates of the final quality in use of different design
solutions can be obtained by testing prototypes with larger numbers of representative users
during development using normalized quality in use measures, and estimating how the resulting
effectiveness, efficiency, and satisfaction adequately mitigate the potential risks.
— Quality assurance and control process: the quality in use of the implemented system can be tested
against requirements.
NOTE 2 Annex E explains the relationship between different SQuaRE quality models.
NOTE 3 In this International Standard, the word “measure” (used as a noun) refers to a quality measure.
6.2 Measurement of quality in use
Quality in use depends not only on the product quality of the software or computer system, but also
on the particular context in which the product is being used (see ISO/IEC 25063). The context of use
includes user factors, task factors, and physical and social environmental factors that can affect quality
in use. Therefore, comparisons of the quality in use of a software product or system are only valid when
the measures are made in the same context of use.
© ISO/IEC 2016 – All rights reserved 7

This makes it particularly difficult to predict or interpret levels of freedom from risk, as it is usually not
possible to control other factors in the context of use that could influence freedom from risk. However,
it is often possible to provide evidence for the potential risks that could result from poor usability or
poor product quality, and to suggest target values for usability or product quality that would mitigate
these risks (see Figure 2). If any measures of usability or product quality do not reach these target
values, the potential impact of the measured values on freedom from risk can be assessed.
EXAMPLE When designing an airline reservation system, a high target level is set for the effectiveness
(success rate) of users booking a flight to the intended destination at the intended time on the intended date to
minimize the likelihood of the potential economic consequences that could arise from any errors.
Many of the measures are defined in a way that allows them to be customized to meet specific needs.
Therefore, meaningful comparisons between measures or with target values can only be made for
measures that have been operationalized in the same way and used in the same or a sufficiently similar
context of use (i.e. with similar types of users carrying out similar tasks in similar environments).
Effectiveness, efficiency, and satisfaction can be assessed by observing representative users carrying
out representative tasks in a realistic context of use (for example, see the methods in Annexes B and C
and ISO/TS 20282-2). The measures can be obtained by simulating a realistic usage environment (for
instance, in a usability laboratory) or by observing operational use of the product. In order to specify
or measure quality in use, it is first necessary to identify each component of the intended context of
use: the users, their goals, and the environment of use. The evaluation needs to be designed to match
this context of use as closely as possible. It is also important that users are only given the type of help
and assistance that would be available to them in the operational environment. These measures can be
used for assurance processes when applying a method such as that specified in ISO/TS 20282-2.
Some external usability measures (ISO/IEC 25023) are tested in a similar way, but evaluate the use of
particular product features during more general use of the product to achieve a typical task as part of a
test of the quality in use.
NOTE 1 Annex B and ISO/TS 20282-2 provide examples of how to measure effectiveness, efficiency, and
satisfaction. Annex A provides examples of how to measure context coverage. See also Reference [19].
NOTE 2 Annex F provides more information on SQuaRE quality measurement concepts.
6.3 Interpretation of quality in use measures
Some quality in use measures (such as the time to complete a task) can be difficult to interpret in
isolation. There are several ways that quality in use measures can be compared so that they are easier
to interpret, namely:
a) conformance: comparing measures with a specific business or usage requirements (e.g. it must be
possible to complete the task in 10 minutes);
b) benchmarks: comparing measures with a benchmark for the same or a similar product or system
used for the same purpose (e.g. it must be possible to complete the tasks with the new system in no
more time than it took with the old system);
c) time series: comparing trends over time (e.g. the reduced number of errors made by users with
each new prototype version of a system);
d) proficiency: comparison with the values obtained when used by a trained or expert user (e.g. how
much longer does it take a new user compared with an experienced user);
e) population norms for satisfaction: when there is a database of previous values, measures can be
expressed as the percentage of users who have previously given a rating of at least this value.
The quality in use measures in Clause 8 include examples that have been normalized in one of these
ways, but for most measures, several forms of normalization are possible, as shown in Annex B.
8 © ISO/IEC 2016 – All rights reserved

Unnormalized measures (such as errors made or task time) can be used to identify specific problems
encountered by individual users, or small groups of users.
To obtain reliable measures for effectiveness, efficiency, or satisfaction, data needs to be obtained from
sufficient users performing tasks to obtain the desired level of statistical confidence that the target
values have been achieved.
NOTE ISO/TS 20282-2 provides an example of a method for summative testing of the effectiveness, efficiency,
and satisfaction of products for use by the general public.
6.4 Selecting appropriate quality in use measures
Factors that can influence the selection of specific quality in use measures include the following:
— the relative importance of effectiveness, efficiency, satisfaction, and freedom from risk;
— specific aspects of effectiveness, efficiency, or satisfaction that could create risks to economic status,
human life, health, or the environment;
— the skills and knowledge required to apply particular measures.
For more information on the process to be used for evaluation of quality in use, see Annex D.
6.5 Aspects of quality in use described in other International Standards
The following International Standards provide further information about specific aspects of quality in
use:
— ISO 9241-11: the definitions of effectiveness, efficiency, and satisfaction in ISO 9241-11 are similar to
those in ISO/IEC 25010, so the quality in use measures for effectiveness, efficiency, and satisfaction
in ISO/IEC 25022 can also be as measures of usability as defined in ISO 9241-11 (see Annex C);
— ISO/IEC 25062 specifies how the results of a summative usability test should be documented;
— ISO/IEC 25063 explains the elements of the context of use that need to be identified when measuring
usability;
— ISO/IEC 25064 explains how user needs (that could include needs for quality in use) should be
documented;
— ISO/TS 20282-2 provides a rigorous methodology for measuring effectiveness, efficiency, and
satisfaction of a consumer product.
7 Format used for documenting the quality measures
The following information is given for each measure in the tables in Clause 8.
a) ID: Identification code of quality measure. Each ID consists of the following three parts:
— code representing the quality characteristics and subcharacteristics. For Effectiveness and
Efficiency use the format “Ef” and “Ey”, and for Satisfaction, Freedom from risk, and Context
coverage, use the format “ABc”, where ”A” represents the characteristic and “Bc” represents the
subcharacteristic;
— serial number of sequential order within quality subcharacteristic;
— G: Generally applicable, could be used in a wide range of situations; S: Specialised for specific
needs;
b) Name: Quality measure name;
© ISO/IEC 2016 – All rights reserved 9

c) Description: The information provided by the quality measure;
d) Measurement function: Formula showing how the quality measure elements are combined to
produce the quality measure;
e) Method: The type of method that can be used to obtain the measure.
— measure user performance: measures of effectiveness and efficiency (see D.3.3.2 and D.3.3.3);
— measure customer behaviour: data collected on actions taken by customers;
— automated data collection: data collected by instrumenting the software to collect user actions;
— questionnaire: measures of satisfaction (see D.3.3.5);
— business analytics: analysis of business activities and results;
— software and usability analysis: analysis of potential risks arising from human or system errors;
— usage statistics: analysis of the consequences arising from previous human or system errors;
— analysis of context description: analysis of the context of use to assess the expected quality in use;
— inspection: inspection of the system to identify potential problems.
8 Quality in use measures
8.1 General
The quality measures in this Clause are listed by quality characteristics and subcharacteristics, in the
order used in ISO/IEC 25010: effectiveness, efficiency, satisfaction, freedom from risk, and context
coverage.
The values obtained for measures of effectiveness, efficiency, and satisfaction depend on the context of
use (see 6.2), so the types of users, tasks, and the environments for which measures are made (or are to
be made) has to be stated in combination with any measurement results.
The need for compliance with standards or regulations can be identified as part of requirements for a
system, but these are outside the scope of the quality model.
NOTE This list of quality measures is not finalized and might be revised in future versions of this
International Standard. Readers of this International Standard are invited to provide feedback.
8.2 Effectiveness measures
Effectiveness measures assess the accuracy and completeness with which users achieve specified goals.
NOTE 1 Effectiveness measures do not take account of how the goals were achieved, only the extent to which
they were achieved (see D.3.1.2).
NOTE 2 The phrase “errors made by the user” refers to the user not performing the intended actions. These
errors are sometimes called “use errors” to emphasise that the main cause of the error can be bad system design.
10 © ISO/IEC 2016 – All rights reserved

Table 1 — Effectiveness measures
ID Name Description Measurement function Method
Ef- Tasks com‑ The proportion of the tasks that X = A/B Measure user
1-G pleted are completed correctly with- performance
A = Number of unique tasks completed
out assistance
B = Total number of unique tasks
attempted
NOTE 1  This measure can be measured for one user or a group of users.
NOTE 2  If tasks can be partially completed the Objectives
...