ISO/TS 20282-2:2013

TECHNICAL ISO/TS
SPECIFICATION 20282-2
Second edition
2013-08-01
Usability of consumer products and
products for public use —
Part 2:
Summative test method
Facilité d’emploi des produits de consommation courante et des
produits à usage public —
Partie 2: Méthode d’essai sommative
Reference number
©
ISO 2013
© ISO 2013
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized otherwise in any form
or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior
written permission. Permission can be requested from either ISO at the address below or ISO’s member body in the country of
the requester.
ISO copyright office
Case postale 56 • CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 749 09 47
E-mail copyright@iso.org
Web www.iso.org
Published in Switzerland
ii © ISO 2013 – All rights reserved

Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Conformity . 2
3 Normative references . 2
4 Terms and definitions . 2
5 General principles . 6
5.1 Type of usage of the product to be tested . 6
5.2 Purpose of the test . 6
5.3 Scope of the goals used in the test . 7
5.4 Validity and reliability . 9
5.5 Ethical issues . 9
6 Test procedure overview . 9
7 Test method .10
7.1 Purpose and scope of the test .10
7.2 Product(s) to be tested .10
7.3 Context of evaluation .10
7.4 Check that the product is capable of achieving the intended goals for the
intended users .15
7.5 Design and perform the test .16
8 Results .20
8.1 General .20
8.2 Goals .20
8.3 Effectiveness .20
8.4 Efficiency .21
8.5 Satisfaction .21
9 Report .21
Annex A (informative) User goals .22
Annex B (informative) Background to the method .25
Annex C (normative) Recruiting a representative sample of users .26
Annex D (normative) Confidence intervals .29
Annex E (normative) Examples of satisfaction scales.33
Annex F (normative) Format for test reports .35
Annex G (normative) Specification of usability requirements .41
Annex H (normative) Usability test procedure specification .44
Annex I (informative) Feedback on the standard .48
Bibliography .49
Foreword
ISO (the International Organization for Standardization) is a worldwide federation of national standards
bodies (ISO member bodies). The work of preparing International Standards is normally carried out
through ISO technical committees. Each member body interested in a subject for which a technical
committee has been established has the right to be represented on that committee. International
organizations, governmental and non-governmental, in liaison with ISO, also take part in the work.
ISO collaborates closely with the International Electrotechnical Commission (IEC) on all matters of
electrotechnical standardization.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for the
different types of ISO documents should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2. www.iso.org/directives
Attention is drawn to the possibility that some of the elements of this document may be the subject of
patent rights. ISO shall not be held responsible for identifying any or all such patent rights. Details of any
patent rights identified during the development of the document will be in the Introduction and/or on
the ISO list of patent declarations received. www.iso.org/patents
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation on the meaning of ISO specific terms and expressions related to conformity
assessment, as well as information about ISO’s adherence to the WTO principles in the Technical
Barriers to Trade (TBT) see the following URL: Foreword - Supplementary information
The committee responsible for this document is ISO/TC 159, Ergonomics, Subcommittee SC 4, Ergonomics
of human-system interaction.
This second edition cancels and replaces the first editions of ISO/TS 20282-2:2006, ISO/PAS 20282-
3:2007, and ISO/PAS 20282-4:2007, of which it constitutes a technical revision. The main changes are
the following:
— clarification that the test method is only intended to be used when there are a limited number of goals to
be tested and it is possible to identify typical contexts of use and criteria for successful goal achievement;
— provision for a wider range of methods to test achievement of goals;
— use of ISO 9241-11 terminology: usability, effectiveness, efficiency, and satisfaction;
— wider range of levels of confidence with a sample size as low as 10, using the Adjusted Wald statistic;
— wider range of purposes for use of the test method.
ISO/TS 20282 consists of the following parts, under the general title Usability of consumer products and
products for public use:
— Part 1: Design requirements for context of use and user characteristics
— Part 2: Summative test method (Technical Specification)
iv © ISO 2013 – All rights reserved

Introduction
Many people find some consumer products and walk-up-and-use products, including consumer products
provided for public use, difficult to install and use, particularly when using them for the first time or at
infrequent intervals. This is clearly undesirable for the producers of such products, for organizations that
use the products to provide a service, and for the people who use them. Information about the usability
of a product would, therefore, be of great value to producers, as part of development and marketing,
to service providers, and to potential purchasers making purchase decisions or comparing alternative
products. This would provide an incentive for producing products that are easier to install and use and
would enable potential purchasers to pay specific attention to usability when selecting a product to buy
and use. It is difficult to judge usability in a purchase situation without available comparable usability
test results.
Usability (see ISO 9241-11) is the extent to which a product can be used by specified users to achieve
specified goals with effectiveness, efficiency, and satisfaction in a specified context of use. Effectiveness
is fundamental as it is about achieving the intended goal(s). Efficiency is about the resources (such as
time or effort) needed by users to achieve their goals so it can be important. In addition, it is important
that users are satisfied with their experience, particularly where users have discretion over whether
to use a product and can readily choose some alternative means of achieving their goals. In this part
of ISO/TS 20282, accessibility is operationalized as the extent to which a product can be used with
effectiveness, efficiency, and satisfaction by people from a population with the widest range of
characteristics and capabilities to achieve a specified goal in a specified context of use.
Poor usability and/or accessibility can result in errors that can lead to several types of risks, for example,
inconvenience resulting from not achieving a goal or achieving the wrong goal, incurring unexpected
costs, or physical injury. In many countries, there are legal requirements to provide accessible products,
services, and facilities.
EXAMPLE Calling the wrong person by mistake with a mobile phone may have the negative consequence of
possible undesirable call charges either for the caller or the person called (who may have to pay for the call).
In addition to the risks of potential adverse consequences for the user as a result of failing to achieve
their goal or achieving the wrong goal (poor effectiveness), there are other risks such as being late
as a result of poor efficiency or users avoiding the use of a difficult-to-use product as a result of poor
satisfaction.
Formative evaluation using expert inspection or user-based testing to provide feedback to improve the
usability of the product is an integral part of the iterative human-centred design process recommended in
ISO 9241-210. Summative evaluation can be used to validate usability and/or accessibility requirements,
to provide a benchmark, or to provide a basis for comparison of different products. Although some
types of expert inspection methods based on a checklist or a standard can provide summative data, the
aspects of usability and/or accessibility that are measured are limited in comparison with the measures
of effectiveness, efficiency, and satisfaction provided by user-based testing.
EXAMPLE One study found that only 50 % of the problems encountered on 16 websites by 32 blind users
[23]
were covered by Success Criteria in the Web Content Accessibility Guidelines 2.0 (WCAG 2.0).
Inspection can precede user-based testing to identify (and, if possible, eliminate) easily identifiable problems
and to check that the product is capable of achieving the intended goals for the intended users (see 7.4).
To provide reliable data on effectiveness, efficiency, and satisfaction that can be compared, it is
desirable to have a standard summative user-based test procedure. This part of ISO/TS 20282 specifies
a summative user-based test method that can be used to provide an evaluation of the usability and/or
accessibility and ease of unpacking, setting up, and installation of consumer products, and the usability
and/or accessibility of products for public use (including walk-up-and-use products). It can be applied to
products that are used to achieve goals that have clear success criteria and relate to well-defined types
of subject matter.
ISO/TS 20282-1 describes in more detail sources of variance in user characteristics that form part of the
context of use that needs to be taken into account when designing for usability. This information is also
needed to identify the elements of the context of use required for testing in this part of ISO/TS 20282.
Further information about the characteristics of older people and people with disabilities can be found
in ISO/TR 22411.
vi © ISO 2013 – All rights reserved

TECHNICAL SPECIFICATION ISO/TS 20282-2:2013(E)
Usability of consumer products and products for public
use —
Part 2:
Summative test method
1 Scope
This part of ISO/TS 20282 specifies a user-based summative test method for the measurement of the
usability and/or accessibility of consumer products and products for public use (including walk-up-and-
use products) for one or more specific user groups. This test method treats accessibility as a special case
of usability where the users taking part in the test represent the extremes of the range of characteristics
and capabilities within the general user population. When the test method refers to usability, the method
can also be used to test accessibility (unless otherwise specified).
This test method is for use when valid and reliable measures of effectiveness, efficiency, and
satisfaction are needed.
NOTE 1 Products for public use include walk-up-and-use products that provide a service to the general public.
The test method can also be used to assess the usability and/or accessibility of achieving the goals of
unpacking, installing, and setting up a consumer product.
This part of ISO/TS 20282 is intended to be used for testing the usability and/or accessibility of
products when
— it is possible to identify typical contexts of use that are representative of the use of the product(s),
— it is possible to identify the criteria for the successful achievement of the users’ goal, and
— there are a limited number of goals being tested at the same time.
While the test method is intended to test consumer products and products for public use, it can also be
used to test other products, systems, and services with the characteristics described above.
If use of a product involves interaction with inputs, outputs, or environments that are highly variable
and/or complex with variability or complexity that cannot be categorized in well-defined subsets, it
is outside the scope as it would not be possible to obtain reliable results. See Annex A for examples of
products and goals that are within the scope of this part of ISO/TS 20282.
EXAMPLE The method could be applied to an office photocopier, a website selling books or train tickets, or a
legal advice service. The method would not be appropriate for a complex ecommerce website, a word processor,
or a bicycle.
The method is primarily intended for use for assessing completed versions of products, but could also be
used for internal purposes during development to judge, assess, and communicate the usability and/or
accessibility of functional prototype versions.
The results of the summative test method can be used for the following purposes:
— to estimate the probability of achieving target values of effectiveness, efficiency, and satisfaction
in actual use;
— to publish information about the usability and/or accessibility of a product;
— to compare the usability and/or accessibility of several products;
— to compare the results with a usability and/or accessibility requirements specification;
— to support procurement.
NOTE 2 Annex H lists the information to be included when specifying the procedure used to test whether the
usability and/or accessibility requirements (Annex G) have been met.
The intended users of this part of ISO/TS 20282 are people with expertise in the design and management
of testing usability and/or accessibility, working within or on behalf of manufacturers, suppliers,
purchasing organizations, or third parties (such as test organizations or consumer organizations).
2 Conformity
A report of the values for the usability and/or accessibility of a product conforms to this part of
ISO/TS 20282 if
— the test method used conforms to the requirements in Clauses 6, 7, 8, and 9 and Annexes C and D, and
— the report of the results contains the information specified in Annex F.
A statement of requirements for usability results conforms to this part of ISO/TS 20282 if it conforms
to the requirements in Annex G.
The specification of a usability test procedure conforms to this part of ISO/TS 20282 if it conforms to
the requirements in Annex H.
3 Normative references
The following documents, in whole or in part, are normatively referenced in this document and are
indispensable for its application. For dated references, only the edition cited applies. For undated
references, the latest edition of the referenced document (including any amendments) applies.
ISO 20282-1, Ease of operation of everyday products — Part 1: Design requirements for context of use and
user characteristics
4 Terms and definitions
For the purposes of this document, the following terms and definitions apply.
4.1
accessibility
extent to which products, systems, services, environments, and facilities can be used by people from
a population with the widest range of characteristics and capabilities to achieve a specified goal in a
specified context of use
Note 1 to entry: Context of use includes direct use or use supported by assistive technologies.
[SOURCE: ISO 26800:2011, definition 2.1]
Note 2 to entry: When carrying out a summative user-based measurement of accessibility, all three components
of usability (effectiveness, efficiency, and satisfaction) need to be considered.
4.2
actual users
group(s) of people who directly interact with a product
Note 1 to entry: Before a product is released, this is the intended user group, and after release this is based on
what is known about the actual user group.
[SOURCE: ISO 20282-1:2006, definition 3.1]
2 © ISO 2013 – All rights reserved

4.3
consumer product
product that is intended to be acquired and used by an individual for personal rather than professional use
[SOURCE: ISO 20282-1:2006, definition 3.2]
4.4
context of evaluation
users, tasks, equipment (hardware, software, and materials), and the physical and social environments
in which a product is evaluated
[SOURCE: ISO/TS 20282-2:2006, definition 4.3]
4.5
context of use
users, tasks, equipment (hardware, software, and materials), and the physical and social environments
in which a product is used
[SOURCE: ISO 9241-11:1998, definition 3.5]
4.6
ease of interaction
usability of interaction with the user interface of a product
Note 1 to entry: Ease of interaction is the effectiveness, efficiency, and satisfaction with which the user can
successfully interact with the interface of the product.
4.7
effectiveness
accuracy and completeness with which users achieve specified goals
[SOURCE: ISO 9241-11:1998, definition 3.2]
4.8
efficiency
resources expended in relation to the accuracy and completeness with which users achieve goals
[SOURCE: ISO 9241-11:1998, definition 3.3]
4.9
formative evaluation
evaluation designed and used to improve the object of evaluation, especially when it is still being developed
[SOURCE: ISO/TR 18152:2010, definition 4.6]
Note 1 to entry: A formative test method is used to perform a formative evaluation.
4.10
goal
intended outcome
[SOURCE: ISO 9241-11:1998, definition 3.8]
Note 1 to entry: A goal is stated independently of the functionality used to achieve it.
4.11
intended users
group(s) of people for whom a product is designed
Note 1 to entry: In many cases, the actual user population is different from that originally intended by the manufacturer.
The intended user group is based on realistic estimations of who the actual users of the product will be.
[SOURCE: ISO 20282-1:2006, definition 4.12]
4.12
interaction
bidirectional information exchange between users and equipment
[SOURCE: IEC/TR 61997:2001, definition 3.4]
Note 1 to entry: Equipment includes both hardware and software.
Note 2 to entry: Information exchange may include physical actions, resulting in sensory feedback.
4.13
main goal(s)
most frequent or important goal(s) that all or a large majority of users want to achieve when using a product
[SOURCE: ISO 20282-1:2006, definition 3.14]
Note 1 to entry: Main goals can depend on achieving sub-goals.
Note 2 to entry: Examples of main goals are given in Annex A.
4.14
satisfaction
freedom from discomfort, and positive attitudes towards the use of the product
[SOURCE: ISO 9241-11:1998, definition 3.4]
4.15
stratified sample
sample established by a procedure in which the population is divided into subpopulations (strata), each
one of which contributes with a specified number of randomly selected individuals
[SOURCE: ISO 15535:2006, definition 3.4]
4.16
success rate
percentage of users successfully achieving a goal
4.17
summative evaluation
evaluation designed to present conclusions about the merit or worth of the object of evaluation
Note 1 to entry: The results can be used to produce recommendations about whether it should be retained,
altered, or eliminated.
Note 2 to entry: It is possible to design a method to provide a combined formative and summative evaluation.
Note 3 to entry: A summative test method is used to perform a summative evaluation.
[SOURCE: ISO/TS 18152:2010, definition 4.10, modified — Reference to recommendations has been
removed and notes to entries have been added.]
4.18
task
activities required to achieve a goal
Note 1 to entry: These activities can be physical and/or cognitive.
[SOURCE: ISO 9241-11:1998, definition 3.9]
4 © ISO 2013 – All rights reserved

4.19
usability
extent to which a system, product, or service can be used by specified users to achieve specified goals
with effectiveness, efficiency, and satisfaction in a specified context of use
[SOURCE: ISO 9241-210:2010, definition 2.13]
Note 1 to entry: References to the term “usability” in the description of the user-based test method include the
application of the test method to accessibility (unless otherwise specified).
4.20
usability requirement
required level of usability expressed in terms of measures of effectiveness, efficiency, and satisfaction
in a specified context of use
4.21
usability testing
evaluation that involves representative users performing specific tasks with the system to enable the
measurement of efficiency, effectiveness, and/or user satisfaction
[SOURCE: ISO/IEC 25060:2010, definition 2.17]
4.22
user
person who interacts with a system, product, or service
Note 1 to entry: The person who uses a service provided by a work system, such as a customer in a shop or
passenger on a train, can be considered a user.
[SOURCE: ISO 26800:2011, 2.10, modified — Notes 1 and 3 have been omitted.]
4.23
user characteristics
attributes of a user that could influence usability
[SOURCE: ISO 20282-1:2006, 3.20, modified — The word “may” has been replaced with “could”.]
4.24
user group
group of users differentiated by characteristics of the users, tasks, or environments that are
expected to influence usability
Note 1 to entry: This could either be an intended user group or a user test group.
4.25
user interface
elements of a product used to control it and receive information about its status
Note 1 to entry: A list of operating instructions permanently displayed on the product is part of the user interface.
EXAMPLE The user interface of a shower tap is the water control lever, where the movement of the lever
controls the temperature of the water and the position of the lever communicates the temperature to the user.
[SOURCE: ISO 20282-1:2006, definition 3.21, modified — The phrase “and the interaction that enables
the user to use it for its intended purpose” has been deleted.]
4.26
user test group
group of persons selected to participate in a test of usability, sampled according to specific requirements
4.27
walk-up-and-use product
product that provides a service to the general public
Note 1 to entry: Walk-up-and-use products are designed to enable users to approach and use the product
successfully without previous experience.
5 General principles
5.1 Type of usage of the product to be tested
This user-based test method can be used to measure the usability and accessibility of
— unpacking, installation, and setting up of consumer products,
— use of consumer products,
— products for public use, including walk-up-and-use products that provide a service to the general
public, and
— other products that are used to achieve goals that have clear success criteria and relate to well-
defined types of subject matter.
NOTE Accessibility is particularly important for walk-up-and-use products and for consumer products
provided for public use.
The test method measures effectiveness, efficiency, and satisfaction when used by specified user
groups in specified contexts of use. Accessibility is measured by the extent to which products, systems,
services, environments, and facilities can be used by people from a population with the widest range of
characteristics and capabilities to achieve a specified goal in a specified context of use.
5.2 Purpose of the test
This part of ISO/TS 20282 can be used by product manufacturers to
a) test a single product to determine whether usability and/or accessibility requirements have been met,
b) test a single product to provide evidence of the usability and/or accessibility of a product for a
customer or for marketing purposes,
c) test a single product to establish a benchmark against which future products can be compared,
d) make comparisons between different products,
e) make comparisons between versions of the same product, and
f) specify usability and/or accessibility requirements for a product to be developed (Annex G) and the
scenarios to use when testing whether the requirements have been met (Annex H).
This part of ISO/TS 20282 can be used for procurement by corporate or public purchasers to
a) test a single product to decide whether it meets usability and/or accessibility requirements,
b) test a group of similar products to make comparisons to facilitate decisions on which is the most suitable,
c) ensure that tests of similar products use the same methodology so that information about usability
and/or accessibility can be compared during the procurement process, and
d) specify usability and/or accessibility requirements for a product to be procured (for example, when
products are to be used in a school, hotel, or homes for the elderly) (Annex G) and the scenarios to
be used when testing whether the requirements have been met (Annex H).
6 © ISO 2013 – All rights reserved

NOTE 1 ISO/IEC 25062 provides a format for reporting this information for professional products.
This part of ISO/TS 20282 can be used by a third-party test organization or consumer advocate
organization to
a) test a single or multiple product(s) to decide whether they meet the usability and/or accessibility
requirements for a user group,
b) test a single or multiple product(s) to establish a benchmark against which future products can be
compared,
c) compare alternative products to provide information to be included in reports, and
d) make comparisons between competing systems (testing the main goals of use of the type of product
or system unless there are specific reasons for testing other goals).
EXAMPLE 1 A testing organization wants to find out how easy it is for new owners of mobile phones to make
a call from the address book. The requirement is to be 80 % confident of an 80 % success rate. A representative
group of 12 people who have not previously owned this brand of mobile phone are selected. To meet the
requirement, 11 out of 12 users will have to be successful within a predefined time limit. Success rate and task
time are recorded, and satisfaction is measured after each task using the system usability scale (SUS).
EXAMPLE 2 A manufacturer wants to demonstrate that a personal video recorder (PVR) is easy to program.
The manufacturer has defined a requirement to be 95 % confident that 80 % of the user population can program
the PVR. A representative sample of 30 people who have purchased a PVR with the intention of programming it
to record programmes is recruited. They are selected to be representative in age, education, and ownership of
different brands of PVR. Each session is expected to take a total of 15 min to 20 min and two PVRs are set up with
TVs for programming. Each person is asked to program the PVR to record a specified programme. The test result
is given as the measured success rate together with a confidence interval. The requirement is met when 28 of the
30 people tested are successful.
EXAMPLE 3 A manufacturer wants to demonstrate that a multifunction home printer/copier is easy to install
for current users of other printers/copiers. The manufacturer has defined a requirement to be 95 % confident that
80 % of the user population can successfully install the printer. A sample of 30 people who own a printer/copier
at home is recruited. They are selected to be representative in age, education, and ownership of different brands.
Each session is expected to take a total of 10 min to 30 min and two kits are available. The printer/copier is
returned to its default settings on completion of each test and repackaged as it would be received by the consumer.
Each person is asked to install the printer/copier and print and copy a test page. The test result is given as the
measured success rate together with a confidence interval and the geometric mean task time together with a
confidence interval.
5.3 Scope of the goals used in the test
5.3.1 Selection of goals
Two decisions need to be made about the scope of the goals used in the test.
a) Which goal(s) are to be tested for achievement in which context(s) of use (see 5.3.2)?
b) Is goal achievement going to be assessed by whether interaction with the interface is successful
(ease of interaction) or by whether the overall outcome resulting from the interaction meets the
needs of the user (overall usability of the product)(see 5.3.3)?
5.3.2 Testing achievement of the main goals in the main context of use or specific goal(s) in
specific context(s) of use
The scope of the test can be to
a) test the achievement of the main goals in the main context(s) of use in order to obtain measures
of effectiveness, efficiency, and satisfaction that are representative of the overall usability or
accessibility of the product. (When testing installation and setup, the main goal is to install the
most common configuration to obtain measures that are representative of the overall usability of
the installation process.)
b) test the achievement of more specific goal(s) and/or using more specific context(s) of use to obtain
measures that are representative of the usability of the product or installation process in specific
context(s) of use (for example, for a specific user group (see 7.3.3), specific environment, or specific task).
5.3.3 Measuring installation and setup, the usability of the user interface, and interaction or
overall usability
The prerequisites for usability of a product (i.e. the prerequisites for users to be effective, efficient, and
satisfied) are
a) the specified users can successfully interact with the interface of the product in specified contexts
of use (ease of interaction), and
b) the product is capable of producing results of acceptable technical quality as a result of the interaction.
While interaction with the product should enable the user to achieve their overall goals, the product also
has to be capable of producing adequate results.
EXAMPLE 1 If the goal is to use a camera to take photographs that can be used for large prints, the user has to
be able to successfully interact with the interface and the quality of the image generated by the camera needs to
be adequate for a large print.
The scope of the test can be
a) unpacking, setting up, and installation of consumer products,
b) usability of the user interface and interaction (“ease of interaction”), or
NOTE If a product reliably produces results of acceptable technical quality, it is sufficient to just test the
ease of interaction.
c) usability of the product as a whole (“overall usability”: the extent to which the product enables the
user to achieve their overall goal with effectiveness, efficiency, and satisfaction).
EXAMPLE 2 The effectiveness of achieving the goal of using an alarm clock to be woken in the morning could
be decomposed.
Scope of test Interactions Goals Assumptions
(success criteria)
Unpacking, setting up, and Unpack the clock and set Clock unpacked, None
installation the current time working, and set to
the correct time
Usability of the user interface Set the alarm hours and Alarm is correctly When the alarm is cor-
and interaction (ease of interac- minutes for a specified set (usability of the rectly set, it will keep time
tion) time and turn on the user interface) and always sound at the
alarm on the clock specified time. Alarm is
sufficiently loud to wake the
user.
Usability of the product as a Not specified Alarm sounds at the None
whole specified time and
wakes the user
The usability of an alarm clock for the goal of being woken in the morning could be tested by finding out
whether, after setting the alarm, the user is woken at the specified time in the morning.
Alternatively, if it is known that when the alarm is correctly set, it will always sound at the specified
time and will be sufficiently loud to wake the user, measuring the usability of interaction with the user
8 © ISO 2013 – All rights reserved

interface by observing whether the user correctly sets the time and turns the alarm on is sufficient to
be confident of the usability of the product for the goal of being woken.
So if the alarm clock is known to have the required technical quality, testing the usability of interaction
with the user interface is also a test of the usability of the product. If the technical quality of the alarm
clock is not known, the usability of interaction with the user interface does not provide a guarantee of
the usability of the alarm clock for waking the user up.
5.4 Validity and reliability
The design of user-based tests should pay particular attention to ensuring that the results will be
— valid, i.e. that the data that have been gathered are, in fact, measures of what the test intended to
measure (i.e. they are appropriate measures of usability), and
— reliable, i.e. that the test would produce consistent results if it were repeated.
A valid and reliable test provides the basis from which one can justify that the results can be generalized
to predict the usability of the tested product.
The test method specified in Clause 6 incorporates good experimental practice designed to maximize
the validity and reliability of the results. It is particularly important that a representative sample of
users takes part in the test if the results are to apply to the intended user population.
5.5 Ethical issues
Any test involving human participants should implement good practice in addressing ethical issues.
NOTE 1 Some countries have human subject research legislation. Many organizations follow established
procedures and regulations. A number of professional organizations have codes of practice for the ethical
governance of experimental work. The aim of such codes of practice is to safeguard the welfare and rights of
those involved, including the participants and other people who could be affected by the test, as well as the
investigators.
In particular, the design and conduct of tests should ensure that the rights and safety of individuals are
protected. For example:
— No test should demand excessive levels of physiological or mental effort.
NOTE 2 This is of particular importance when tests involve participation by older people or people with
disabilities.
— The privacy of the individuals involved should be safeguarded.
— The confidentiality of any data collected should be maintained.
— If some user groups are thought to be vulnerable, for example, children, older people, or people with
disabilities, special care should be taken to establish that their rights have been protected.
— Participants should be provided with sufficient information about the purposes and conduct of any
study to enable them to decide whether to participate or not on the basis of this information.
— Participants should be entitled to withdraw from participation at any stage.
6 Test procedure overview
To measure usability and/or accessibility using the summative test method in Clause 7, the following
activities shall be carried out.
NOTE As there are interdependencies, some of the activities may need to take place in parallel.
a) Define the purpose and scope of the test (see 7.1).
b) Identify the product(s) to be tested (see 7.2).
c) Specify the goals to be included in the test (see 7.3.2).
d) Specify the user group(s) to be used for testing (see 7.3.3).
e) Consider whether to use the selection of user test groups to assess accessibility (see 7.3.3.4).
f) Specify environmental characteristics (see 7.3.4).
g) Check that product is compatible with the user characteristics and goals (see 7.4).
h) Identify the relevant measures (see 7.5.2).
i) Determine the number of participants required for the desired confidence level (see 7.5.3).
j) Recruit a sample of users (see 7.5.4).
k) Configure the product (see 7.5.5).
l) Specify test scenarios and goal criteria (see 7.5.6).
m) Establish the test procedure (see 7.5.7).
n) Create the test environment (see 7.5.8).
o) Assess whether the criteria for goal achievement have been met by individual users (see 7.5.10).
p) Calculate the results (see Clause 8).
q) Prepare a full report and, if required, a short summary (see Clause 9 and Annex F).
7 Test method
7.1 Purpose and scope of the test
The type of usage of the product to be tested shall be identified and, whether testing for accessibility is
included, shall be decided (see 7.3.3.4). The purpose of the test (see 5.2) and scope (see 5.3) of the test
shall be defined to enable an appropriate test to be designed that provides the required information.
7.2 Product(s) to be tested
Identify the specific product(s) and version(s) to be tested and, if appropriate, the organization
responsible for the provision of the service supported by the product.
If any of the products to be tested do not have product certification, it will be necessary to ensure that
they are safe for the participants to use.
EXAMPLE Issues could include electrical safety, sharp edges, or small loose parts when testing with children.
7.3 Context of evaluation
7.3.1 General
The context used for evaluation shall be specified. This context and the goals to be achieved should be as
close as possible to the actual or intended goals and context of use for specific user group(s).
NOTE 1 More information about the context of use can be found in ISO/IEC 25063.
10 © ISO 2013 – All rights reserved

NOTE 2 If detailed information is not already available about the current or intended context of use, this will
have to be obtained. For example, if a test is being carried out by or on behalf of a manufacturer, the manufacturer
might already have this information.
7.3.2 Specify how goal achievement will be me
...