ISO/IEC 4944:2024

International
Standard
ISO/IEC 4944
First edition
Information technology — User
2024-09
interfaces — Evaluating usability of
natural user interfaces
Technologie de l'information — Interfaces utilisateur — Cadre
pour l'évaluation de l'utilisabilité d'une interface utilisateur
naturelle
Reference number
© ISO/IEC 2024
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on
the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below
or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
© ISO/IEC 2024 – All rights reserved
ii
Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
3.1 Terms related to natural user .1
3.2 Terms related to usability .2
4 Natural user interfaces (NUI). 2
5 Evaluating usability of a natural user interface (NUI) . 3
5.1 Framework .3
5.2 NUI as a system, product or service being evaluated .4
5.3 Evaluating usability for diverse NUI users .4
5.4 Evaluating usability for goals, tasks and interactions of NUI.5
5.5 Evaluating usability for diverse contexts of use of NUI .5
5.6 Components of usability.6
5.7 Evaluation based on use .6
6 Measuring usability of a NUI . 7
6.1 Purposes of and measures for evaluation .7
6.2 Measures for evaluating the user interface of NUI .8
6.2.1 Evaluating user interface of NUI for accomplishing tasks .8
6.2.2 Effectiveness measures .8
6.2.3 Efficiency measures.9
6.2.4 Satisfaction measures . .10
6.3 Measures for evaluating naturalness of interactions with NUI.10
6.3.1 Evaluating naturalness of interactions with NUI for accomplishing tasks .10
6.3.2 Effectiveness measures .10
6.3.3 Efficiency measures.11
6.3.4 Satisfaction measures . . 12
7 Reporting usability evaluation results .12
7.1 Content elements for reporting usability evaluations . 12
7.2 Format for reporting usability evaluations . 12
Bibliography .13

© ISO/IEC 2024 – All rights reserved
iii
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical activity.
ISO and IEC technical committees collaborate in fields of mutual interest. Other international organizations,
governmental and non-governmental, in liaison with ISO and IEC, also take part in the work.
The procedures used to develop this document and those intended for its further maintenance are described
in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for the different types
of document should be noted. This document was drafted in accordance with the editorial rules of the ISO/
IEC Directives, Part 2 (see www.iso.org/directives or www.iec.ch/members_experts/refdocs).
ISO and IEC draw attention to the possibility that the implementation of this document may involve the
use of (a) patent(s). ISO and IEC take no position concerning the evidence, validity or applicability of any
claimed patent rights in respect thereof. As of the date of publication of this document, ISO and IEC had not
received notice of (a) patent(s) which may be required to implement this document. However, implementers
are cautioned that this may not represent the latest information, which may be obtained from the patent
database available at www.iso.org/patents and https://patents.iec.ch. ISO and IEC shall not be held
responsible for identifying any or all such patent rights.
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and expressions
related to conformity assessment, as well as information about ISO's adherence to the World Trade
Organization (WTO) principles in the Technical Barriers to Trade (TBT) see www.iso.org/iso/foreword.html.
In the IEC, see www.iec.ch/understanding-standards.
This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 35, User interfaces.
Any feedback or questions on this document should be directed to the user’s national standards
body. A complete listing of these bodies can be found at www.iso.org/members.html and
www.iec.ch/national-committees.

© ISO/IEC 2024 – All rights reserved
iv
Introduction
As smart devices such as smart phones and smart speakers have become popular, new forms of user
interfaces have been introduced. One aspect of the new interfaces is mimicking natural ways of interacting
with the devices and utilizing speech, gestures, body language, eye gaze and other natural features of
human communication through natural user interfaces (NUIs), which allow for faster, smoother, intuitive,
and smarter interaction. However, while the concept of NUI is well defined, practical applications of NUI still
need to be developed. This is strongly related to the lack of usability of NUI in smart devices.
When specified users of systems, products or services achieve their specified goals with effectiveness,
efficiency and satisfaction in a specified context of use (as defined in ISO/IEC 9241-11), usability can be
evaluated. Effectiveness is considered as the ability of users to complete tasks with the system (e.g. the NUI).
Efficiency measures resources taken by users to complete specified tasks. Satisfaction measures comfort
levels of user experience along with acceptance.
It is important that the usability evaluation of NUI concentrates on recognizing the user's intention and
focuses on multiple modalities of information flows between the users and the devices.
This document describes a framework for evaluating the usability of NUI. Although the framework can be
applied in most general contexts, it is specifically focused on optimizing user experiences in terms of NUI.

© ISO/IEC 2024 – All rights reserved
v
International Standard ISO/IEC 4944:2024(en)
Information technology — User interfaces — Evaluating
usability of natural user interfaces
1 Scope
This document provides a framework, requirements and recommendations for evaluating the usability of
a natural user interface (NUI) for systems, products or services. Measuring and reporting the usability of
the NUI are also described. The usability evaluation focuses on the efficiency, effectiveness and satisfaction
of the NUI. This document can also be applied to the usability evaluation of other user interfaces (UIs) for
emerging technologies.
2 Normative references
The following documents are referred to in the text in such a way that some or all of their content constitutes
requirements of this document. For dated references, only the edition cited applies. For undated references,
the latest edition of the referenced document (including any amendments) applies.
ISO 9241-11, Ergonomics of human-system interaction — Part 11: Usability: Definitions and concepts
3 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO 9241-11 and the following apply.
ISO and IEC maintain terminology databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at https:// www .electropedia .org/
3.1 Terms related to natural user
3.1.1
user interface
UI
set of all the components of an interactive system that provide information and controls for the user to
accomplish specific tasks with the interactive system
[SOURCE: ISO 9241-110:2020, 3.10]
3.1.2
natural user interface
NUI
user interface where the user operates the interface through a manner related to everyday human
behaviours in a real world
Note 1 to entry: Natural refers to the user's expectations, behaviour and feelings that are involved in interacting with a NUI.
Note 2 to entry: Everyday human behaviours refer to behaviours that the user can be expected to have already learned.
Note 3 to entry: Interactions with NUIs typically model interactions performed in the natural world, including (but not
limited to) touch, gestures or vocal interactions, or a combination thereof.

© ISO/IEC 2024 – All rights reserved
3.1.3
natural user interface interaction
NUI interaction
user-system interaction making use of a natural user interface
3.1.4
naturalness
correspondence with everyday interactions between humans and the real-world objects with which they
interact
Note 1 to entry: Naturalness relates to everyday behaviours that the user can be expected to have already learned.
EXAMPLE To perform a “hello” gesture, the user refers to an object by pointing; to order something the user
issues a vocal command; or to agree the user nods in the affirmative or voices a vocal onomatopoeia for agreement.
3.2 Terms related to usability
3.2.1
interaction
user-system interaction
exchange of information between a user and an interactive system via the user interface to complete the
intended task
Note 1 to entry: Adapted from ISO 9241-110:2020, 3.11 definition of user-system interaction.
3.2.2
diverse users
individuals with differing abilities and characteristics or accessibility needs
[SOURCE: ISO/IEC Guide 71:2014, 2.3]
3.2.3
diverse contexts
differing contexts of use and differing economic, cultural and organizational conditions
[SOURCE: ISO/IEC Guide 71:2014, 2.8]
4 Natural user interfaces (NUI)
Natural user interfaces (NUIs) are a special case of user interfaces that focus on the naturalness of
interactions that they support.
The evaluation of the usability of an NUI includes:
a) evaluation of the usability of the user interface to accomplish tasks;
NOTE 1 This is similar to the evaluation of any other user interface.
b) evaluation of the usability of the naturalness of the user interface.
NOTE 2 This involves an additional, more specialized evaluation that is described in this document.
The naturalness of the user interface is provided by the NUI interactions, which are recognized and used by
the NUI.
NUI interactions attempt to maximize their achievement of the interaction principles identified in
ISO 9241-110 as:
— Suitability for the user’s tasks: an interactive system is suitable for the user's tasks when it supports the
users in the completion of their tasks, i.e. when the operating functions and the user-system interactions
are based on the task characteristics.

© ISO/IEC 2024 – All rights reserved
— Self-descriptiveness: the interactive system presents appropriate information, where needed by the
user, to make its capabilities and use immediately obvious to the user without unnecessary user-system
interactions.
— Conformity with user expectations: the interactive system’s behaviour is predictable based on the
context of use and commonly accepted conventions in this context.
— Learnability: the interactive system supports discovery of its capabilities and how to use them, allows
exploration of the interactive system, minimizes the need for learning and provides support when
learning is needed.
— Controllability: the interactive system allows the user to maintain control of the user interface and the
interactions, including the speed and sequence and individualization of the user-system interaction.
— Use error robustness: the interactive system assists the user in avoiding use errors and, in case of
identifiable use errors, treats them tolerantly and assists the user when recovering from the errors.
— User engagement: the interactive system presents functions and information in an inviting and motivating
manner supporting continued interaction with the system.
NUI maximizes the naturalness of its interactions by supporting the following:
— Suitability for the user's tasks: the users of the interactive system with NUI perform tasks in a reasonable
manner to how they would perform in the real world with everyday interactions (if the computer was
not involved).
— Self-descriptiveness: the interactive system with NUI presents appropriate information to the user in a
manner that is related to how the information exists in the real world (rather than in a manner that is
optimized for the computer).
— Conformity with user expectations: the interactive system with NUI recognises and processes predictable
user interactions in a predictable manner that is consistent with the real world (rather than designed
based on typical human-computer interaction styles).
— Learnability: the interactive system with NUI supports the users in the discovery and exploration of
its capabilities and information (without the need for formal training regarding the user interface and
supported interactions before they can use the system).
— Controllability: the interactive system with NUI allows the user to interact with the NUI in the manner
and sequence that is most natural to their preferred manner (rather than in a single prescribed manner).
— Use error robustness: the interactive system with NUI allows the users to try different interactions while
avoiding and minimizing any resulting use errors and by providing suitable error recovery where needed.
— User engagement: the interactive system with NUI invites and motivates the users to interact with the
NUI in a similar manner to everyday human behaviour in the real world.
5 Evaluating usability of a natural user interface (NUI)
5.1 Framework
Usability as defined in ISO 9241-11 is a complex concept.
NOTE Usability is one of the outcomes of use of a system, product, or service. There are other outcomes of use, e.g.
accessibility, user experience, avoidance from harm from use. This document only focuses on usability.
Different users will experience different levels of usability of the same system, product or service in
achieving different goals in different contexts of use. Therefore, evaluation of the usability involves a number
of different more specific evaluations that are described in the following subclauses.

© ISO/IEC 2024 – All rights reserved
Evaluating usability of NUI involves applying the definition of usability, as illustrated in Figure 1, by
considering the following three issues:
— the system, product or service being evaluated (see 5.2), which in this case is the NUI;
— the users, goals, tasks and context of use (see 5.3, 5.4 and 5.5) which determine how NUI will be evaluated
and used;
— the components of usability evaluation (see 5.6), e.g. effectiveness, efficiency and satisfaction.
The evaluation of the NUI usability requires evaluation metrics as follows:
a) how to evaluate usability of NUI (see 5.7), which includes:
1) evaluating the usability of the user interface of the NUI for accomplishing goals;
2) evaluating the usability of the naturalness of the interactions supported by the NUI;
b) how to measure the components of usability of NUI (see Clause 6);
c) how to report the results of the usability evaluation (see Clause 7).
Figure 1 — Framework for evaluating usability of NUI
5.2 NUI as a system, product or service being evaluated
While usability can be applied to many different things, it is important that usability evaluation focuses on a
well-defined object of the evaluation.
While NUI is a part of a system that exists within a context of use, it is important that usability evaluation of
NUI is limited to the context of use of the interface and does not extend to the whole system.
NUI can involve multiple media and modalities. It is important to clarify whether the evaluation is focusing
on individual components of the NUI or the entire NUI.
5.3 Evaluating usability for diverse NUI users
In order for NUI to be natural, it is important that it be usable by a wide range of users.
When establishing the specified users for evaluating the usability of NUI, it is important to consider the
usability of the NUI for diverse users.

© ISO/IEC 2024 – All rights reserved
Distinct user groups that together represent the set of diverse users should be identified as the specified
users for the purposes of the usability evaluation of NUI.
5.4 Evaluating usability for goals, tasks and interactions of NUI
Traditional usability focuses on goals and the various tasks that can be used to achieve these goals. This
focus limits the evaluation of usability to specific outcomes.
While evaluating usability, NUI still needs to be limited to consideration of specific outcomes. It needs to be
open to more diverse approaches to achieving these intended outcomes than the typical set of activities that
are used to describe a single task.
In NUI, users are likely to try the set of interactions that they find to be most natural to use. While the
system will recogn
...