ISO/IEC 25022:2016

DRAFT INTERNATIONAL STANDARD
ISO/IEC DIS 25022.2
ISO/IEC JTC 1/SC 7 Secretariat: SCC
Voting begins on: Voting terminates on:
2015-08-03 2015-10-03
Systems and software engineering — Systems and
software quality requirements and evaluation (SQuaRE) —
Measurement of quality in use
Titre manque
ICS: 35.080
THIS DOCUMENT IS A DRAFT CIRCULATED
FOR COMMENT AND APPROVAL. IT IS
THEREFORE SUBJECT TO CHANGE AND MAY
NOT BE REFERRED TO AS AN INTERNATIONAL
STANDARD UNTIL PUBLISHED AS SUCH.
IN ADDITION TO THEIR EVALUATION AS
BEING ACCEPTABLE FOR INDUSTRIAL,
TECHNOLOGICAL, COMMERCIAL AND
USER PURPOSES, DRAFT INTERNATIONAL
STANDARDS MAY ON OCCASION HAVE TO
BE CONSIDERED IN THE LIGHT OF THEIR
POTENTIAL TO BECOME STANDARDS TO
WHICH REFERENCE MAY BE MADE IN
Reference number
NATIONAL REGULATIONS.
ISO/IEC DIS 25022.2:2015(E)
RECIPIENTS OF THIS DRAFT ARE INVITED
TO SUBMIT, WITH THEIR COMMENTS,
NOTIFICATION OF ANY RELEVANT PATENT
RIGHTS OF WHICH THEY ARE AWARE AND TO
©
PROVIDE SUPPORTING DOCUMENTATION. ISO/IEC 2015

ISO/IEC DIS 25022.2:2015(E)
© ISO/IEC 2015, Published in Switzerland
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized otherwise in any form
or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior
written permission. Permission can be requested from either ISO at the address below or ISO’s member body in the country of
the requester.
ISO copyright office
Ch. de Blandonnet 8 • CP 401
CH-1214 Vernier, Geneva, Switzerland
Tel. +41 22 749 01 11
Fax +41 22 749 09 47
copyright@iso.org
www.iso.org
ii © ISO/IEC 2015 – All rights reserved

Contents
1 Scope . 8
2 Conformance . 9
3 Normative references . 9
4 Terms and definitions . 9
5 Abbreviated terms .13
6 Use of quality in use measures .13
6.1 Applications of quality in use .13
6.2 Measurement of quality in use .13
6.3 Normalising measures .14
6.4 Selecting appropriate quality in use measures .14
6.5 Aspects of quality in use described in other International Standards .15
7 Format used for documenting the quality measures.15
8 Quality in use measures .16
8.1 General .16
8.2 Effectiveness measures.16
8.3 Efficiency measures .17
8.4 Satisfaction measures .18
8.4.1 General .18
8.4.2 Usefulness measures .19
8.4.3 Trust measures .20
8.4.4 Pleasure measures .20
8.4.5 Comfort measures .20
8.5 Freedom from risk measures .20
8.5.1 General .20
8.5.2 Economic risk mitigation measures .21
8.5.3 Health and safety risk mitigation measures .22
8.5.4 Environmental risk mitigation measures .23
8.6 Context coverage measures .23
8.6.1 General .23
8.6.2 Context completeness measures .23
8.6.3 Flexibility measures .24
Annex A (Informative) Examples of how to measure context coverage .25
Annex B (Informative) Normalisation of quality in use measures .27
Annex C (Informative) Use of ISO/IEC 25022 for measuring usability in ISO 9241-11 .31
Annex D (Informative) Quality in use evaluation process .32
D.1 Establish evaluation requirements .32
D.2 Specify the evaluation .32
D.3 Design the evaluation .35
D.4 Execute the evaluation .35
Annex E (Informative) Relationship between different quality models .36
Annex F (Informative) Quality measurement concepts .37
Annex G (Informative) QMEs used to define quality measures .38
Bibliography .39
© ISO/IEC 2015 – All rights reserved
ISO/IEC DIS 25022.2(E)
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical Commission)
form the specialized system for worldwide standardization. National bodies that are members of ISO or IEC
participate in the development of International Standards through technical committees established by the
respective organization to deal with particular fields of technical activity. ISO and IEC technical committees
collaborate in fields of mutual interest. Other international organizations, governmental and non-governmental, in
liaison with ISO and IEC, also take part in the work. In the field of information technology, ISO and IEC have
established a joint technical committee, ISO/IEC JTC 1.
International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2.
The main task of the joint technical committee is to prepare International Standards. Draft International Standards
adopted by the joint technical committee are circulated to national bodies for voting. Publication as an International
Standard requires approval by at least 75 % of the national bodies casting a vote.
Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights.
ISO and IEC shall not be held responsible for identifying any or all such patent rights.
ISO/IEC 25022, which replaces ISO/IEC 9126-4, is a part of the SQuaRE series of standards and was prepared by
Joint Technical Committee ISO/IEC JTC 1, information technology, Subcommittee SC 7, Software and Systems
Engineering
The SQuaRE series of standards consists of the following divisions under the general title Systems and software
Quality Requirements and Evaluation:
 ISO/IEC 2500n - Quality Management Division,
 ISO/IEC 2501n - Quality Model Division,
 ISO/IEC 2502n - Quality Measurement Division,
 ISO/IEC 2503n - Quality Requirements Division,
 ISO/IEC 2504n - Quality Evaluation Division,
 ISO/IEC 25050 - 25099 SQuaRE Extension Division.
Annexes A, B, C, D, E, F and G are for information only.
© ISO/IEC 2015 – All rights reserved 5

Introduction
This International Standard is a part of the SQuaRE series of international standards. It provides a set of measures
for the characteristics of quality in use (defined in ISO/IEC 25010) that can be used for specifying quality in use
requirements (in conjunction with ISO/IEC 25030) and measuring and evaluating quality in use (in conjunction with
ISO/IEC 25040 and ISO/IEC 25041).
The quality measures included in this International Standard were selected based on their practical value. They are
based on established practice (including for example [4] in the Bibliography), They are not intended to be
exhaustive, and users of this International Standard are encouraged to refine them if necessary.
This International Standard replaces ISO/IEC 9126-4, and has the following changes:
 Measures are given for the revised quality model for quality in use in ISO/IEC 25010.
 Measures are categorised as generally applicable, could be used in a wide range of situations or
specialised for specific needs.
 Annexes that were common to ISO/IEC 9126-2, -3 and -4 have been removed (and might be included in a
future revision of ISO/IEC 25020).
Quality Measurement Division
This International Standard is a part of ISO/IEC 2502n Quality Measurement Division of SQuaRE series that
currently consists of the following International Standards:
 ISO/IEC 25020 – Measurement reference model and guide: provides a reference model and guide for
measuring the quality characteristics defined in ISO/IEC 2501n Quality Model Division.
 ISO/IEC 25021 – Quality measure elements: provides a format for specifying Quality Measure Elements
and some examples of QMEs that can be used to construct software quality measures.
 ISO/IEC 25022 – Measurement of quality in use: provides measures, including associated measurement
functions for the quality characteristics in the quality in use model.
 ISO/IEC 25023 – Measurement of system and software product quality: provides measures, including
associated measurement functions and QMEs for the quality characteristics in the product quality model.
 ISO/IEC 25024 – Measurement of data quality: provides measures, including associated measurement
functions and QMEs for the quality characteristics in the data quality model.
Figure 1 depicts the relationship between this International Standard and the other standards in ISO/IEC 2502n
division.
© ISO/IEC 2015 – All rights reserved
ISO/IEC DIS 25022.2(E)
Figure 1 - Structure of the Quality Measurement division

Outline and Organization of SQuaRE Series
The SQuaRE series consists of five main divisions and extension division. Outline of each divisions within SQuaRE
series are as follows.
 ISO/IEC 2500n - Quality Management Division. The standards that form this division define all common
models, terms and definitions referred further by all other standards from SQuaRE series. The division also
provides requirements and guidance for the planning and management of a project.
 ISO/IEC 2501n - Quality Model Division. The standards that forms this division provides quality models for
system/software products, quality in use and data. A service quality model is under development. Practical
guidance on the use of the quality model is also provided.
 ISO/IEC 2502n - Quality Measurement Division. The standards that form this division include a
system/software product quality measurement reference model, definitions of quality measures, and
practical guidance for their application. This division presents internal measures of software quality,
external measures of software quality and quality in use measures. Quality measure elements forming
foundations for the quality measures are defined and presented.
 ISO/IEC 2503n - Quality Requirements Division. The standard that forms this division helps specifying
quality requirements. These quality requirements can be used in the process of quality requirements
elicitation for a system/software product to be developed, designing a process for achieving necessary
quality, or as inputs for an evaluation process.
 ISO/IEC 2504n - Quality Evaluation Division. The standards that form this division provide requirements,
recommendations and guidelines for system/software product evaluation, whether performed by
independent evaluators, acquirers or developers. The support for documenting a measure as an Evaluation
Module is also presented.
ISO/IEC 25050 to ISO/IEC 25099 are reserved for SQuaRE extension International Standards, which currently
include 25051 Requirements for quality of Ready to Use Software Products (RUSP) and instructions for testing,
and the ISO/IEC 25060 – 25069 Common industry format for usability series of standards.
© ISO/IEC 2015 – All rights reserved 7

Systems and software engineering – Systems and software Quality
Requirements and Evaluation (SQuaRE) – Measurement of quality in use
1 Scope
This International Standard defines quality in use measures for the characteristics defined in ISO/IEC 25010, and is
intended to be used together with ISO/IEC 25010. It can be used in conjunction with the ISO/IEC 2503n – Quality
requirements division and the ISO/IEC 2504n – Quality evaluation division standards or to more generally meet
user needs with regard to product or system quality.
This International Standard contains:
 a basic set of measures for each quality in use characteristic;
 an explanation of how quality in use is measured.
This International Standard provides a suggested set of quality in use measures to be used with the quality in use
model in ISO/IEC 25010. They are not intended to be an exhaustive set.
It includes as informative annexes examples of how to measure context coverage (Annex A), options for
normalising quality in use measures (Annex B), Use of ISO/IEC 25022 for measuring usability in ISO 9241-11,
(Annex C), a quality in use evaluation process (Annex D), the relationship between different quality models (Annex
E) and quality measurement concepts (Annex F).
The measures are applicable to the use of any human-computer system, including both computer systems in use
and software products that form part of the system.
This International Standard does not assign ranges of values of the measures to rated levels or to grades of
compliance, because these values are defined for each system, product depending on the context of use and
users' needs.
Some attributes could have a desirable range of values, which does not depend on specific user needs but
depends on generic factors; for example, human cognitive factors.
The proposed quality in use measures are primarily intended to be used for quality assurance and management of
systems and software products based on their effects when actually used. The main users of the measurement
results are people managing development, acquisition, evaluation or maintenance of software and systems.
The main users of this International Standard are people carrying out specification and evaluation activities as part
of:
 Development: including requirements analysis, design, and testing through acceptance during the life cycle
process.
 Quality management: systematic examination of the product or computer system, for example when
evaluating quality in use as part of quality assurance and quality control.
 Supply: a contract with the acquirer for the supply of a system, software product or software service under
the terms of a contract, for example when validating quality at qualification test.
 Acquisition: including product selection and acceptance testing, when acquiring or procuring a system,
software product or software service from a supplier.
 Maintenance: improvement of the product based on quality in use measures.
© ISO/IEC 2015 – All rights reserved
ISO/IEC DIS 25022.2(E)
2 Conformance
Any quality requirement specification or quality evaluation that conforms to this International Standard shall:
a) Select the quality in use characteristics and/or subcharacteristics to be specified or evaluated as defined in
ISO/IEC 25010.
b) For each selected characteristic or subcharacteristic, select all the General (G) quality measures defined in
clause 8 or provide a rationale for any that are excluded.
c) Optionally select any other quality measures that are relevant.
d) If any quality measure is modified, provide the rationale for any changes.
e) Define precisely how each quality measure is operationalised (for example details of the measurement
method or questionnaire used).
f) Define any additional quality measures used that are not included in this International Standard.
3 Normative references
The following documents, in whole or in part, are normatively referenced in this document and are indispensable
for its application. For dated references, only the edition cited applies. For undated references, the latest edition of
the referenced document (including any amendments) applies.
ISO/IEC 25010:2011, Systems and software engineering - Systems and software Quality Requirements and
Evaluation (SQuaRE) - System and software quality models
4 Terms and definitions
For the purposes of this International Standard, the following terms and definitions and those given in ISO/IEC
25000 and in ISO/IEC 25010 apply.
NOTE: The essential definitions from ISO/IEC 25000 SQuaRE series and the other ISO standards are reproduced here.
4.1
context completeness
degree to which a product or system can be used with the required levels of effectiveness, efficiency, satisfaction
and freedom from risk in each of the specified contexts of use
[SOURCE: ISO/IEC 25010:2011 modified – added "the required levels of" and changed "all" to "each of" for
clarification]
Note 1 to entry: Context completeness is a subcharacteristic of context coverage.
4.2
context coverage
degree to which a product or system can be used with effectiveness, efficiency, satisfaction and freedom from risk
in both specified contexts of use and in contexts beyond those initially explicitly identified
Note 1 to entry: Context of use is relevant to both quality in use and some product quality (sub)characteristics (where it is
referred to as “specified conditions”).
[SOURCE: ISO/IEC 25010:2011 4.1.5]
4.3
effectiveness
accuracy and completeness with which users achieve specified goals
[SOURCE: ISO 9241-11:1998]
4.4
efficiency
resources expended in relation to the accuracy and completeness with which users achieve goals
[SOURCE: ISO 9241-11:1998]
© ISO/IEC 2015 – All rights reserved 9

Note 1 to entry: Relevant resources can include time to complete the task (human resources), materials, or the financial cost of
usage.
4.5
context of use
users, tasks, equipment (hardware, software and materials), and the physical and social environments in which a
system, product or service is used
[SOURCE: ISO 9241-11:1998, 3.5, with "product" replaced by "system, product or service".]
4.6
flexibility
degree to which a product or system can be used with acceptable levels of effectiveness, efficiency, freedom from
risk in contexts beyond those initially specified in the requirements satisfaction and
[SOURCE: ISO/IEC 25010:2011, modified – added "acceptable levels of" for clarification.]
Note 1 to entry: Flexibility is a subcharacteristic of context coverage.
4.7
formative evaluation
evaluation designed and used to improve the object of evaluation, especially when it is still being developed
[SOURCE: ISO/TS 18152, 2010, 4.6]
4.8
freedom from risk
degree to which the quality of a product or system mitigates or avoids potential risks to economic status, human life,
health, or the environment
Note 1 to entry: Risk is a function of the probability of occurrence of a given threat and the potential adverse consequences of
that threat's occurrence.
[SOURCE: ISO/IEC 25010:2011, modified – added "quality of" and "or avoids" for clarification]
Note 2 to entry: The risks considered by the SQuaRE series are those arising from insufficient product quality.
Note 3 to entry: Freedom from risk includes reduction of potential risks to the user, organisation or project.
4.9
goal
intended outcome
[SOURCE: ISO 9241-11:1998]
4.10
measure (noun)
variable to which a value is assigned as the result of measurement
Note 1 to entry: The term “measures” is used to refer collectively to base measures, derived measures, and indicators.
[SOURCE: ISO/IEC 15939:2007]
Note 2 to entry: In this document when the word "measure" is used qualified by a characteristic or subcharacteristic it refers to
a quality measure.
4.11
measurement
set of operations having the object of determining a value of a measure
[SOURCE: ISO/IEC 15939:2007]
Note 1 to entry: Measurement can include assigning a qualitative category such as the language of a source program (ADA, C,
COBOL, etc.).
© ISO/IEC 2015 – All rights reserved
ISO/IEC DIS 25022.2(E)
4.12
measurement function
algorithm or calculation performed to combine two or more quality measurement elements
[SOURCE: ISO/IEC 25021:2012]
4.13
psychometrics
field of study concerned with the theory and technique for developing valid and reliable psychological measures
4.14
quality in use
degree to which a product or system can be used by specific users to meet their needs to achieve specific goals
with effectiveness, efficiency, satisfaction and freedom from risk in specific contexts of use
[SOURCE: ISO/IEC 25010: 2011]
Note 1 to entry: The quality in use of a software product or system can be measured and evaluated by the effect of the target
system or software products when used by users of the implemented system, or during field testing or prototype testing.
Note 2 to entry: When quality in use is specified it relates to specified users meeting their needs to achieve specified goals with
effectiveness, efficiency, satisfaction and freedom from risk in specified contexts of use.
4.15
quality measure
measure that is defined as a measurement function of two or more values of quality measure elements
[SOURCE: ISO/IEC 25021:2012]
4.16
quality measure element
measure defined in terms of a property and the measurement method for quantifying it, including optionally the
transformation by a mathematical function
[SOURCE: ISO/IEC 25021:2012]
4.17
quality model
defined set of characteristics, and of relationships between them, which provides a framework for specifying quality
requirements and evaluating quality
[SOURCE: ISO/IEC 25000:2014]
4.18
satisfaction
degree to which user needs are satisfied when a product or system is used in a specified context of use
Note 1 to entry: For a user who does not directly interact with the product or system, only purpose accomplishment and trust
are relevant.
Note 2 to entry: Satisfaction is the user’s response to interaction with the product or system, and includes attitudes towards use
of the product.
[SOURCE: ISO/IEC 25010:2011]
Note 3 to entry: Users include: primary users who interacts with the system to achieve the primary goals, secondary users who
provide support and indirect users who receives output, but does not interact with the system.
Note 4 to entry: In this International Standard, user's needs include their desires and expectations associated with use of a
product, system or service. Exceeding desires and expectations is a means of significantly increasing satisfaction and improving
the user experience.
© ISO/IEC 2015 – All rights reserved 11

4.19
stakeholder satisfaction
degree to which stakeholder needs are satisfied when a product or system is used in a specified context of use
[SOURCE: ISO/IEC 25010:2011 definition for the term "satisfaction" modified to refer to stakeholders]
Note 1 to entry: Users of a product or system are one type of stakeholder, so user satisfaction is one type of stakeholder
satisfaction.
4.20
summative evaluation
evaluation designed to present conclusions about the merit or worth of the object of evaluation
Note 1 to entry: The results can be used to produce recommendations about whether it should be retained, altered or
eliminated.
Note 2 to entry: It is possible to design a method to provide a combined formative and summative evaluation.
Note 3 to entry: A summative test method is used to perform a summative evaluation.
[SOURCE: ISO TS 20282-2:2012, 4.16]
4.21
system
a combination of interacting elements organised to achieve one or more stated purposes
Note 1 to entry: A system may be considered as a product or as the services it provides.
Note 2 to entry: In practice, the interpretation of its meaning is frequently clarified by the use of an associative noun, e.g.
aircraft system. Alternatively the word system may be substituted simply by a context dependent synonym, e.g. aircraft, though
this may then obscure a system principles perspective.
[SOURCE: ISO/IEC 15288:2008]
4.22
task
activities required to achieve a goal
Note 1 to entry: These activities can be physical or cognitive.
Note 2 to entry: Job responsibilities can determine goals and tasks.
[SOURCE: ISO 9241-11:1998]
4.23
usability
degree to which a product or system can be used by specified users to achieve specified goals with effectiveness,
efficiency and satisfaction in a specified context of use
Note 1 to entry: Based on ISO 9241-210.
Note 2 to entry: Usability can either be specified or measured as a product quality characteristic in terms of its
subcharacteristics, or specified or measured directly by measures that are a subset of quality in use.
[SOURCE: ISO/IEC 25010:2011]
4.24
use error
act or omission of an act that results in a different system response than intended by the manufacturer or expected
by the user
[SOURCE: IEC 62366:2007 with "medical device" replaced by "system"]
© ISO/IEC 2015 – All rights reserved
ISO/IEC DIS 25022.2(E)
4.25
user
individual or group that benefits from a system during its utilization
[SOURCE: ISO/IEC 15939:2007]
5 Abbreviated terms
The following abbreviation is used in this International Standard.
 QME – Quality Measure Element
6 Use of quality in use measures
6.1 Applications of quality in use
Quality in use is the degree to which a product or system can be used by specific users to meet their needs to
achieve specific goals with effectiveness, efficiency and satisfaction and reduced risk in specific contexts of use.
NOTE 1 Quality in use is thus a combination of usability (when specified or measured directly by measures of effectiveness,
efficiency and satisfaction) and freedom from risk.
Quality in use measures the outcomes of interaction between a user and a system. The final quality in use can be
measured when an implemented system is used in the user’s environment for its intended purpose. Quality in use
measures can also be used at conceptual design and development stages.
 Quality in use requirements can be specified early in the design and development process, to give a high
level goal for the intended quality to be experienced by users and other stakeholders.
 Specific problems with quality in use can be identified by testing prototypes with small numbers of
representative users during development using unnormalised measures in order to improve the product
(formative evaluation).
 Estimates of the final quality in use of different design solutions can be obtained by testing prototypes with
larger numbers of representative users during development (summative evaluation using normalised
measures), and estimating how the resulting effectiveness, efficiency and satisfaction adequately mitigate
the potential risks.
 The quality in use of the implemented system can be tested against requirements (for quality assurance
and control).
NOTE 2 Annex D explains the relationship between different SQuaRE quality models.
NOTE 3 In this International Standard, the word “measure” (used as a noun) refers to a quality measure.
6.2 Measurement of quality in use
Quality in use depends not only on the software or computer system, but also on the particular context in which the
product is being used (see ISO/IEC 25063). The context of use includes user factors, task factors and physical and
social environmental factors that can affect quality in use. So comparisons of the quality in use of a software
product or system are only valid when the measures are made in the same context of use.
This makes it particularly difficult to interpret measures of freedom from risk, as it is usually not possible to control
other factors in the context of use that could influence freedom from risk. However it is often possible to provide
evidence for the potential risks that could result from poor usability, and to suggest target values for usability that
would mitigate these risks (see Figure 2 in 8.5.1). If any measures of effectiveness, efficiency, satisfaction and
context completeness do not reach these target values when the product or system is used, the potential impact of
the measured values on freedom from risk can be assessed.
EXAMPLE When designing an airline reservation system, a high target level is set for the effectiveness (success rate) of
users booking a flight to the intended destination at the intended time on the intended date, to minimise the likelihood of the
potential economic consequences that could arise from any errors.
Many of the measures are defined in a way that allows them to be customised to meet specific needs. So
meaningful comparisons between measures or with target values can only be made for measures that have been
© ISO/IEC 2015 – All rights reserved 13

operationalised in the same way and used in the same or a sufficiently similar context of use (i.e. with similar types
of users carrying out similar tasks in similar environments).
Effectiveness, efficiency and satisfaction can be assessed by observing representative users carrying out
representative tasks in a realistic context of use (for example see the methods in Annex B and C and ISO 20282-2).
The measures can be obtained by simulating a realistic usage environment (for instance in a usability laboratory) or
by observing operational use of the product. In order to specify or measure quality in use it is first necessary to
identify each component of the intended context of use: the users, their goals, and the environment of use. The
evaluation needs to be designed to match this context of use as closely as possible. It is also important that users
are only given the type of help and assistance that would be available to them in the operational environment.
These measures can be used for assurance processes when applying a method such as that specified in ISO
20282-2.
Some external usability measures (ISO/IEC 25023) are tested in a similar way, but evaluate the use of particular
product features during more general use of the product to achieve a typical task as part of a test of the quality in
use.
NOTE 1 Annex B and ISO 20282-2 provide examples of how to measure effectiveness, efficiency and satisfaction. Annex A
provides examples of how to measure context coverage. See also the Bibliography [6].
NOTE 2 Annex E provides more information on SQuaRE quality measurement concepts.
6.3 Normalising measures
Some quality in use measures (such as the time to complete a task) can be difficult to interpret in isolation. There
are several ways that quality in use measures can be normalised so that they are easier to interpret:
a) Conformance: comparing measures with a specific business or usage requirements (e.g. it must be
possible to complete the task in 10 minutes).
b) Benchmarks: comparing measures with a benchmark for the same or a similar product or system used for
the same purpose (e.g. it must be possible to complete the tasks with the new system in no more time than
it took with the old system).
c) Time series: comparing trends over time (e.g. the reduced number of errors made by users with each new
prototype version of a system).
d) Proficiency: Comparison with the values obtained when used by a trained or expert user (e.g. how much
longer does it take a new user compared with an experienced user).
e) Population norms for satisfaction: when there is a database of previous values, measures can be
expressed as the percentage of users who have previously given a rating of at least this value.
The quality in use measures in clause 8 include examples that have been normalised in one of these ways, but for
most measures several forms of normalisation are possible, as shown in Annex B.
Unnormalised measures (such as errors made or task time) can be used to identify specific problems encountered
by individual users, or small groups of users.
To obtain reliable measures for effectiveness, efficiency or satisfaction, data needs to be obtained from sufficient
users performing tasks to obtain the desired level of statistical confidence that the target values have been
achieved.
NOTE ISO TS 20282-2 provides an example of a method for summative testing of the effectiveness, efficiency and
satisfaction of products for use by the general public.
6.4 Selecting appropriate quality in use measures
Factors that can influence the selection of specific quality in use measures include:
 The relative importance of effectiveness, efficiency, satisfaction and freedom from risk.
 Specific aspects of effectiveness, efficiency or satisfaction that could create risks to economic status,
human life, health, or the environment.
 The skills and knowledge required to apply particular measures.
For more information on evaluation of quality in use, see Annex E.
© ISO/IEC 2015 – All rights reserved
ISO/IEC DIS 25022.2(E)
6.5 Aspects of quality in use described in other International Standards
The following International Standards provide further information about specific aspects of quality in use:
ISO 9241-11: the definitions of effectiveness, efficiency and satisfaction in ISO 9241-11 are similar to those in
ISO/IEC 25010, so the quality in use measures for effectiveness, efficiency and satisfaction in ISO/IEC 25022
can also be as measures of usability as defined in ISO 9241-11 (see Annex C).
ISO/IEC 25062 specifies how the results of a summative usability test should be documented.
ISO/IEC 25063 explains the elements of the context of use that need to be identified when measuring usability.
ISO/IEC 25064 explains how user needs (that could include needs for quality in use) should be documented.
ISO TS 20282-2 provides a rigorous methodology for measuring effectiveness, efficiency and satisfaction of a
consumer product.
7 Format used for documenting the quality measures
The following information is given for each measure in the tables in clause 8.
a) ID: Identification code of quality measure. Each ID consists of the following three parts:
 code representing the quality characteristics and subcharacteristics. For Effectiveness and Efficiency use
the format "Ef" and "Ey", and for Satisfaction, freedom from Risk, and Context coverage use the format
"ABc", where ”A” represents the characteristic and “Bc” represents the subcharacteristic;
 serial number of sequential order within quality subcharacteristic.
 G: Generally applicable, could be used in a wide range of situations; S: Specialised for specific needs.
b) Name: Quality measure name.
c) Description: The information provided by the quality measure.
d) Measurement function: Formula showing how the quality measure elements are combined to produce the
quality measure.
e) Method: The type of method that can be used to obtain the measure.
 Measure user performance: measures of effectiveness and efficiency (see D.2.3.2 and D.2.3.3).
 Measure customer behaviour: data collected on actions taken by customers.
 Automated data collection: data collected by instrumenting the software to collect user actions.
 Questionnaire: measures of satisfaction (see D.2.3.5).
 Business analytics: analysis of business activities and results.
 Software and usability analysis: analysis of potential risks arising from human or system errors.
 Usage statistics: analysis of the consequences arising from previous human or system errors.
 Analysis of context description: analysis of the context of use to assess the expected quality in use.
 Inspection: inspection of the system to identify potential problems.
© ISO/IEC 2015 – All rights reserved 15

8 Quality in use measures
8.1 General
The quality measures in this clause are listed by quality characteristics and subcharacteristics, in the order used in
ISO/IEC 25010: effectiveness, efficiency, satisfaction, freedom from risk and context coverage.
The values obtained for measures of effectiveness, efficiency and satisfaction depend on the context of use (see
6.2), so the types of users, tasks and the environments for which measures are made (or are to be made) has to be
stated in combination with any measurement results.
The need for compliance with standards or regulations can be identified as part of requirements for a system, but
these are outside the scope of the quality model.
NOTE 1 This list of quality measures is not finalised, and might be revised in future versions of this International Standard.
Readers of this International Standard are invited to provide feedback.
NOTE 2 In the quality in use model, the term usability refers to the subset of quality in use composed of effectiveness,
efficiency and satisfaction. Measures of the usability and functional suitability of the user interface and interaction are contained
in ISO/IEC 25023.
8.2 Effectiveness measures
Effectiveness measures assess the accuracy and completeness with which users achieve specified goals.
NOTE 1 Effectiveness measures do not take account of how the goals were achieved, only the extent to which they were
achieved (see D.2.1.2).
NOTE 2 The phrase "errors made by the user" refers to the user not performing the intended actions. These errors are
sometimes called "use errors" to emphasise that the main cause of the error can be bad system design.
Table 1 — Effectiveness measures
ID Name Description Measurement function Method
Ef-1- Tasks The proportion of the tasks that are X = A/B Measure user
completed
G completed correctly without performance
A = Number of unique tasks completed
assistance
B = Total number of unique tasks
attempted
NOTE 1 This measure can be measured for one user or a group of users.
NOTE 2 If tasks can be partially completed the Objectives achieved measure is more appropriate.
NOTE 3 If the tasks are of different complexity, weighted tasks could be used in the formula: X =  (i=1.n) Wi x Ai / B
where i is the number of the task and wi represents the difficulty of that task where total sum of Wi = 1.0.
NOTE 4 This could be applied either to the tasks identified in the requirements or to the tasks attempted by the user.
Ef-2- Objectives The proportion of the objectives of Measure user
{X = 1-A | X>0}
i
achieved
S the task that are achieved correctly performance
A = Proportional value of each missing or
i
without assistance
incorrect objective in the task output
(maximum value = 1)
NOTE Each potential missing or incomplete component is given a weight Ai based on the extent to which it detracts from the value of the
output to the business or user. (If the sum of the weights exceeds 1, the quality measure is normally set to 0, although this can indicate negative
outcomes and excessive risks.) The scoring scheme is refined iteratively by applying it to a series of task outputs and adjusting the weights until
the results obtained are repeatable, reproducible and meaningful.
EXAMPLE: The business impact of potential diary and contact information errors was discussed with several potential customers, leading to the
following scoring scheme for calculating mean goal achievement:
• Installation: all components successfully installed: 100%; for each necessary subcomponent omitted from the installation deduct 20%.
• New contact: all details entered correctly: 100%; for each missing item of information, deduct 50%; for each item of information in the
wrong field, deduct 20%; for each typo deduct 5%.
• New meeting: all details entered correctly: 100%, incorrect time or date: 0%; for each item of information in the wrong field, deduct 20%;
for each typo deduct 5%.
Combined deductions equalling or exceeding 100% would be as scored 0% goal achievement.
© ISO/IEC 2015 – All rights reserved
ISO/IEC DIS 25022.2(E)
ID Name Description Measurement function Method
Ef-3- Errors in a task The number of errors made by the X = A Measure user
G user during a task performance
A = Number of errors made by the user
during a task
NOTE 1 The number of errors made by the user can include all errors, or only uncorrected errors, or only errors that result in the task not being
completed correctly.
NOTE 2 Measures of counts of errors can be used to make comparisons between the same task carried out in different circumstances, for
example when comparing different versions of a system under development.
NOTE 3 To compare errors made in different tasks, the number of errors could be related to the number of actions in each task.
NOTE 4 It is only appropriate to make comparisons if errors have equal importance, or are weighted.
NOTE 5 Errors can be analysed using a user by problem matrix indicating how many users had which problem, in which combination.
Ef-4- Tasks with Proportion of tasks where errors X = A/B Measure user
G errors were made by the user performance
A = Number of tasks with errors

B = Total number of tasks
NOTE The notes to Ef-3-G apply.
Ef-5- Task error Proportion of users making an error X = A/B Measure user
G intensity performance
A = Number of users making an error
B= Total number of users performing the
task
NOTE The notes to Ef-3-G apply.
8.3 Efficiency measures
Efficiency measures assess the resources expended in relation to the accuracy and completeness with which
users achieve goals.
NOTE 1 The most common resource is time to complete the task, although other relevant re
...