SIST-TS CEN/CLC ISO/IEC/TS 12791:2025

SLOVENSKI STANDARD
01-februar-2025
Informacijska tehnologija - Umetna inteligenca - Obravnava neželene
pristranskosti pri nalogah strojnega učenja klasifikacije in regresije (ISO/IEC TS
12791:2024)
Information technology - Artificial intelligence - Treatment of unwanted bias in
classification and regression machine learning tasks (ISO/IEC TS 12791:2024)
Informationstechnik - Künstliche Intelligenz - Behandlung von unerwünschtem Bias bei
Klassifizierungs- und Regressionsaufgaben des maschinellen Lernens (ISO/IEC TS
12791:2024)
Technologies de l'information - Intelligence artificielle - Traitement des biais indésirables
dans les tâches d'apprentissage automatique de classification et de régression (ISO/IEC
TS 12791:2024)
Ta slovenski standard je istoveten z: CEN/CLC ISO/IEC/TS 12791:2024
ICS:
35.020 Informacijska tehnika in Information technology (IT) in
tehnologija na splošno general
SIST-TS CEN/CLC ISO/IEC/TS en,fr,de
12791:2025
2003-01.Slovenski inštitut za standardizacijo. Razmnoževanje celote ali delov tega standarda ni dovoljeno.

TECHNICAL SPECIFICATION CEN/CLC ISO/IEC/TS
SPÉCIFICATION TECHNIQUE
TECHNISCHE SPEZIFIKATION
November 2024
ICS 35.020
English version
Information technology - Artificial intelligence - Treatment
of unwanted bias in classification and regression machine
learning tasks (ISO/IEC TS 12791:2024)
Technologies de l'information - Intelligence artificielle Informationstechnik - Künstliche Intelligenz -
- Traitement des biais indésirables dans les tâches Behandlung von unerwünschtem Bias bei
d'apprentissage automatique de classification et de Klassifizierungs- und Regressionsaufgaben des
régression (ISO/IEC TS 12791:2024) maschinellen Lernens (ISO/IEC TS 12791:2024)
This Technical Specification (CEN/TS) was approved by CEN on 12 October 2024 for provisional application.

The period of validity of this CEN/TS is limited initially to three years. After two years the members of CEN and CENELEC will be
requested to submit their comments, particularly on the question whether the CEN/TS can be converted into a European
Standard.
CEN and CENELEC members are required to announce the existence of this CEN/TS in the same way as for an EN and to make the
CEN/TS available promptly at national level in an appropriate form. It is permissible to keep conflicting national standards in
force (in parallel to the CEN/TS) until the final decision about the possible conversion of the CEN/TS into an EN is reached.

CEN and CENELEC members are the national standards bodies and national electrotechnical committees of Austria, Belgium,
Bulgaria, Croatia, Cyprus, Czech Republic, Denmark, Estonia, Finland, France, Germany, Greece, Hungary, Iceland, Ireland, Italy,
Latvia, Lithuania, Luxembourg, Malta, Netherlands, Norway, Poland, Portugal, Republic of North Macedonia, Romania, Serbia,
Slovakia, Slovenia, Spain, Sweden, Switzerland, Türkiye and United Kingdom.

CEN-CENELEC Management Centre:
Rue de la Science 23, B-1040 Brussels
© 2024 CEN/CENELEC All rights of exploitation in any form and by any means
Ref. No. CEN/CLC ISO/IEC/TS 12791:2024 E
reserved worldwide for CEN national Members and for
CENELEC Members.
Contents Page
European foreword . 3

European foreword
This document (CEN/CLC ISO/IEC/TS 12791:2024) has been prepared by Technical Committee
ISO/IEC JTC 1 "Information technology" in collaboration with Technical Committee CEN-CENELEC/ JTC
21 “Artificial Intelligence” the secretariat of which is held by DS.
Attention is drawn to the possibility that some of the elements of this document may be the subject of
patent rights. CEN-CENELEC shall not be held responsible for identifying any or all such patent rights.
Any feedback and questions on this document should be directed to the users’ national standards
body/national committee. A complete listing of these bodies can be found on the CEN and CENELEC
websites.
According to the CEN-CENELEC Internal Regulations, the national standards organizations of the
following countries are bound to announce this Technical Specification: Austria, Belgium, Bulgaria,
Croatia, Cyprus, Czech Republic, Denmark, Estonia, Finland, France, Germany, Greece, Hungary, Iceland,
Ireland, Italy, Latvia, Lithuania, Luxembourg, Malta, Netherlands, Norway, Poland, Portugal, Republic of
North Macedonia, Romania, Serbia, Slovakia, Slovenia, Spain, Sweden, Switzerland, Türkiye and the
United Kingdom.
Endorsement notice
The text of ISO/IEC TS 12791:2024 has been approved by CEN-CENELEC as
Technical
Specification
ISO/IEC TS 12791
First edition
Information technology — Artificial
2024-10
intelligence — Treatment of
unwanted bias in classification and
regression machine learning tasks
Technologies de l'information — Intelligence artificielle —
Traitement des biais indésirables dans les tâches d'apprentissage
automatique de classification et de régression
Reference number
ISO/IEC TS 12791:2024(en) © ISO/IEC 2024

ISO/IEC TS 12791:2024(en)
© ISO/IEC 2024
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on
the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below
or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
© ISO/IEC 2024 – All rights reserved
ii
ISO/IEC TS 12791:2024(en)
Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
3.1 General .1
3.2 Artificial intelligence .3
3.3 Bias .4
3.4 Testing .5
4 Abbreviated terms . 6
5 Treating unwanted bias in the AI system life cycle . 6
5.1 Inception .6
5.1.1 Stakeholder identification .6
5.1.2 Stakeholder needs and requirements definition .7
5.1.3 Procurement .8
5.1.4 Data sources .9
5.1.5 Integration with risk management .11
5.1.6 Acceptance criteria .11
5.2 Design and development . 12
5.2.1 Feature representation . 12
5.2.2 Metadata sufficiency . 12
5.2.3 Data annotations . 12
5.2.4 Adjusting data . 13
5.2.5 Methods for managing identified risks . 13
5.3 Verification and validation . 13
5.3.1 General . 13
5.3.2 Static testing of data used in development .14
5.3.3 Dynamic testing .14
5.4 Re-evaluation, continuous validation, operations and monitoring . 15
5.4.1 General . 15
5.4.2 External change .16
5.5 Disposal . . .17
6 Techniques to address unwanted bias . 17
6.1 General .17
6.2 Algorithmic and training techniques .17
6.2.1 General .17
6.2.2 Pre-trained models .18
6.3 Data techniques .19
7 Handling bias in a distributed AI system life cycle . 19
Annex A (informative) Life cycle processes map .21
Annex B (informative) Potential impacts of unwanted bias on different types of specific user .22
Bibliography .23

© ISO/IEC 2024 – All rights reserved
iii
ISO/IEC TS 12791:2024(en)
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical activity.
ISO and IEC technical committees collaborate in fields of mutual interest. Other international organizations,
governmental and non-governmental, in liaison with ISO and IEC, also take part in the work.
The procedures used to develop this document and those intended for its further maintenance are described
in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for the different types
of documents should be noted. This document was drafted in accordance with the editorial rules of the ISO/
IEC Directives, Part 2 (see www.iso.org/directives or www.iec.ch/members_experts/refdocs).
ISO and IEC draw attention to the possibility that the implementation of this document may involve the
use of (a) patent(s). ISO and IEC take no position concerning the evidence, validity or applicability of any
claimed patent rights in respect thereof. As of the date of publication of this document, ISO and IEC had not
received notice of (a) patent(s) which may be required to implement this document. However, implementers
are cautioned that this may not represent the latest information, which may be obtained from the patent
database available at www.iso.org/patents and https://patents.iec.ch. ISO and IEC shall not be held
responsible for identifying any or all such patent rights.
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and expressions
related to conformity assessment, as well as information about ISO's adherence to the World Trade
Organization (WTO) principles in the Technical Barriers to Trade (TBT) see www.iso.org/iso/foreword.html.
In the IEC, see www.iec.ch/understanding-standards.
This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 42, Artificial intelligence], in collaboration with the European Committee for
Standardization (CEN) Technical Committee CEN/CLC/JTC 21, Artificial Intelligence, in accordance with the
Agreement on technical cooperation between ISO and CEN (Vienna Agreement).
Any feedback or questions on this document should be directed to the user’s national standards
body. A complete listing of these bodies can be found at www.iso.org/members.html and
www.iec.ch/national-committees.

© ISO/IEC 2024 – All rights reserved
iv
ISO/IEC TS 12791:2024(en)
Introduction
This document describes steps that can be taken to treat unwanted bias during the development or use of AI
systems.
This document is based on ISO/IEC TR 24027 and provides treatment techniques in accordance with the AI
system life cycle as defined in ISO/IEC 22989:2022, Clause 6 and ISO/IEC 5338. The treatment techniques
in this document are agnostic of context. This document is based on the types of bias described in
ISO/IEC TR 24027.
This document describes good practises for treating unwanted bias and can help an organization with the
treatment of unwanted bias in machine learning (ML) systems that conduct classification and regression
tasks. The techniques in this document are applicable to classification and regression ML tasks. This
document does not address applicability of the described methods outside of the defined ML tasks.
This document does not contain organizational management and enabling processes related to an AI
management system, which can be found in ISO/IEC 42001.
Annex A provides a cross-reference between the life cycle stages and the clauses of this document.

© ISO/IEC 2024 – All rights reserved
v
Technical Specification ISO/IEC TS 12791:2024(en)
Information technology — Artificial intelligence — Treatment
of unwanted bias in classification and regression machine
learning tasks
1 Scope
This document describes how to address unwanted bias in AI systems that use machine learning to conduct
classification and regression tasks. This document provides mitigation techniques that can be applied
throughout the AI system life cycle in order to treat unwanted bias. This document is applicable to all types
and sizes of organization.
2 Normative references
The following documents are referred to in the text in such a way that some or all of their content constitutes
requirements of this document. For dated references, only the edition cited applies. For undated references,
the latest edition of the referenced document (including any amendments) applies.
ISO/IEC 5259-4:2024, Artificial intelligence — Data quality for analytics and machine learning (ML) — Part 4:
Data quality process framework
ISO/IEC 22989:2022, Information technology — Artificial intelligence — Artificial intelligence concepts and
terminology
ISO/IEC/IEEE 29119-3:2021, Software and systems engineering — Software testing — Part 3: Test
documentation
3 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO/IEC 22989:2022 and the
following apply.
ISO and IEC maintain terminology databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at https:// www .electropedia .org/
3.1 General
3.1.1
authoritative record
record which possess the characteristics of authenticity, reliability , integrity and useability
[SOURCE: ISO 30300:2020, 3.2.3]
3.1.2
consumer vulnerability
state in which an individual can be placed at risk of harm during their interaction with or a decision by a
service provider due to the presence of personal, situational and market environment factors
[SOURCE: ISO 22458:2022, 3.5, modified — added reference to a decision by a service provider.]

© ISO/IEC 2024 – All rights reserved
ISO/IEC TS 12791:2024(en)
3.1.3
current operating conditions
conditions under which an AI system is currently operating
Note 1 to entry: Conditions can include resource usage, environmental factors, geographic location of use, time of use,
training provided to operators and the target population.
3.1.4
data subject
person to whom data refer
[SOURCE: ISO 25237:2017, 3.18]
3.1.5
data quality model
defined set of characteristics which provides a framework for specifying data quality requirements and
evaluating data quality
[SOURCE: ISO/IEC 25012:2008, 4.6]
3.1.6
disposition
range of records processes associated with implementing records retention, destruction or transfer
decisions which are documented in disposition authorities (3.1.7) or other instruments
[SOURCE: ISO 30300:2020, 3.4.8]
3.1.7
disposition authority
instrument that defines the disposition (3.1.6) actions that are authorized or required for specified records
[SOURCE: ISO 30300:2020, 3.5.4]
3.1.8
intended operating conditions
conditions under which an AI system is meant to function
Note 1 to entry: Conditions can include resource usage, environmental factors, geographic location of use, time of use,
training provided to operators and the target population.
3.1.9
management system
set of interrelated or interacting elements of an organization (3.1.10) to establish policies and objectives, as
well as processes to achieve those objectives
Note 1 to entry: A management system can address a single discipline or several disciplines.
Note 2 to entry: The management system elements include the organization’s structure, roles and responsibilities,
planning and operation.
[SOURCE: ISO/IEC 42001:2023, 3.4]
3.1.10
organization
person or group of people that has its own functions with responsibilities, authorities and relationships to
achieve its objectives
Note 1 to entry: The concept of organization includes, but is not limited to, sole-trader (sole proprietor), company,
corporation, firm, enterprise, authority, partnership, charity or institution or part or combination thereof, whether
incorporated or not, public or private.
Note 2 to entry: If the organization is part of a larger entity, the term “organization” refers only to the part of the larger
entity that is within the scope of the AI management system (3.1.9).

© ISO/IEC 2024 – All rights reserved
ISO/IEC TS 12791:2024(en)
[SOURCE: ISO/IEC 42001:2023, 3.1]
3.1.11
records process
set of activities for managing authoritative records
[SOURCE: ISO 30300:2020, 3.4.13]
3.1.12
user
individual or group that interacts with a system or benefits from a system during its utilization
[SOURCE: ISO/IEC/IEEE 15288:2023, 3.53, modified — Note 1 to entry has been removed.]
3.2 Artificial intelligence
3.2.1
data quality
characteristic of data that the data meet the organization's (3.1.10) data requirements for a specified context
[SOURCE: ISO/IEC 5259-1:2024, 3.4]
3.2.2
data quality characteristic
category of data quality attributes that bears on data quality (3.2.1)
[SOURCE: ISO/IEC 5259-1:2024, 3.5]
3.2.3
data quality measure
variable to which a value is assigned as the result of measurement of a data quality characteristic (3.2.2)
[SOURCE: ISO/IEC 5259-1:2024, 3.7]
3.2.4
data provenance
provenance
information on the place and time of origin, derivation or generation of a data set, proof of authenticity of the
data set, or a record of past and present ownership of the data set
[SOURCE: ISO/IEC 5259-1:2024, 3.16]
3.2.5
extreme data
type of sample that is an outlier with respect to the real-world distribution
3.2.6
feature
measurable property of an object or event with respect to a set of characteristics
Note 1 to entry: Features play a role in training and prediction.
Note 2 to entry: Features provide a machine-readable way to describe the relevant objects. As the algorithm will not
go back to the objects or events themselves, feature representations are designed to contain information the algorithm
is expected to need.
[SOURCE: ISO/IEC 23053:2022, 3.3.3, modified — Clarification of Note 2 to entry has been added.]

© ISO/IEC 2024 – All rights reserved
ISO/IEC TS 12791:2024(en)
3.2.7
functional correctness
degree to which a product or system provides the correct results with the needed degree of precision
Note 1 to entry: AI systems, and particularly those using machine learning methods, do not usually provide functional
correctness in all observed circumstances.
[SOURCE: ISO/IEC 25059:2023, 3.2.3]
3.2.8
intended use
use in accordance with information provided with an AI system, or, in the absence of such information, by
generally understood patterns of usage
[SOURCE: ISO/IEC Guide 51:2014, 3.6, modified — “a product or system” has been changed to “an AI system”.]
3.2.9
inter-annotator agreement
degree of consensus or similarity among the annotations made by different annotators on the same data
3.3 Bias
3.3.1
AI subject
organization, person or entity that is affected by an AI system, service or product
3.3.2
automation bias
propensity for humans to favour suggestions from automated decision-making systems and to ignore
contradictory information from non-automated sources, even if it is correct
[SOURCE: ISO/IEC TR 24027:2021, 3.2.1, modified — “made without automation” was changed to “from non-
automated sources”.]
3.3.3
coverage bias
type of data bias (3.3.4) that occurs when a population represented in a dataset does not match the
population that the machine learning model is making predictions about
3.3.4
data bias
data properties that if unaddressed lead to AI systems that perform better or worse for different objects,
people or groups (3.3.5)
[SOURCE: ISO/IEC TR 24027:2021, 3.2.7]
3.3.5
group
subset of objects in a domain that are linked because they have shared characteristics
[SOURCE: ISO/IEC TR 24027:2021, 3.2.8]
3.3.6
human cognitive bias
bias that occurs when humans are processing and interpreting information
Note 1 to entry: human cognitive bias influences judgement and decision-making.
[SOURCE: ISO/IEC TR 24027:2021, 3.2.4]

© ISO/IEC 2024 – All rights reserved
ISO/IEC TS 12791:2024(en)
3.3.7
representativeness
qualitative assessment of degree to which a given dataset's properties approximate the statistical properties
of the target population (3.3.10) of interest
Note 1 to entry: Representativeness can be quantified through the use of one or more measures pertaining to the size,
distribution or composition of the data.
Note 2 to entry: Representative test data enables verification that an AI system achieves an acceptable level of
functional correctness (3.2.7) for the target population (3.3.10).
Note 3 to entry: Representative training data can enable training a machine learning model that achieves an acceptable
level of functional correctness (3.2.7) for the target population (3.3.10).
3.3.8
selection bias
type of data bias (3.3.4) that can occur when a dataset’s samples are not collected in a way that is
representative of their real-world distribution
3.3.9
statistical bias
type of consistent numerical offset in an estimate relative to the true underlying value
Note 1 to entry: The offset is inherent to most estimates
[SOURCE: ISO 20501:2019, 3.3.9, modified — “inherent to most estimates” was moved to Note 1 to entry.]
3.3.10
target population
group (3.3.5) of AI subjects (3.3.1) that the AI system will process data in relation to
Note 1 to entry: The target population can include organizations or other objects.
3.3.11
at-risk group
subset of stakeholders that can be adversely affected by unwanted bias
Note 1 to entry: at-risk groups can also emerge from intersections of groups as described in ISO/IEC TR 24027.
Note 2 to entry: unforeseen at-risk groups can emerge due to the use of AI systems, as described in 5.1.5.
3.4 Testing
3.4.1
dynamic testing
testing (3.4.8) in which a test item (3.4.5) is evaluated by executing it
[SOURCE: ISO/IEC/IEEE 29119-2:2021, 3.3]
3.4.2
model testing
testing (3.4.8) in which the behaviour of a model is examined against a set of qualities or other criteria
Note 1 to entry: Model testing is usually performed by executing the model on a systematic set of inputs and evaluating
how well its outputs achieve some measure of task performance, such as matching canonical answers or being rated
highly by humans.
3.4.3
static testing
testing (3.4.8) in which a test item (3.4.5) is examined against a set of quality or other criteria without the
test item being executed
[SOURCE: ISO/IEC/IEEE 29119-1:2022, 3.20, modified — The example has been removed.]

© ISO/IEC 2024 – All rights reserved
ISO/IEC TS 12791:2024(en)
3.4.4
test completion report
test summary report
report that provides a summary of the testing (3.4.8) that was performed
[SOURCE: ISO/IEC/IEEE 29119-3:2021, 3.9]
3.4.5
test item
test object
work product to be tested
[SOURCE: ISO/IEC/IEEE 29119-1:2022, 3.107, modified — The example has been removed.]
3.4.6
test objective
reason for performing testing (3.4.8)
[SOURCE: ISO/IEC/IEEE 29119-2:2021, 3.49, modified — The example has been removed.]
3.4.7
test plan
detailed description of test objectives (3.4.6) to be achieved and the means and schedule for achieving them,
organized to coordinate testing (3.4.8) activities for some test item (3.4.5) or set of test items
[SOURCE: ISO/IEC/IEEE 29119-2:2021, 3.50, modified — the Notes to entry have been removed.]
3.4.8
testing
set of activities conducted to facilitate discovery and evaluation of properties of a test item (3.4.5)
Note 1 to entry: Testing activities include planning, preparation, execution, reporting and management activities
insofar as they are directed towards testing.
[SOURCE: ISO/IEC/IEEE 29119-1:2022, 3.131]
4 Abbreviated terms
AI artificial intelligence
ETL extract, transform and load
IID independent and identically distributed
ML machine learning
PII personally identifiable information
5 Treating unwanted bias in the AI system life cycle
5.1 Inception
5.1.1 Stakeholder identification
Stakeholder identification shall be conducted throughout the development of an AI system.
NOTE This identification enables the collection and assessment of information from those stakeholders on
common effects of unwanted bias that they usually experience from AI systems.

© ISO/IEC 2024 – All rights reserved
ISO/IEC TS 12791:2024(en)
Any individuals, groups or organizations who can be foreseeably positively or negatively affected by
unwanted bias in the AI system should be considered, not just AI actors, but also including AI subjects
directly using or gaining benefit from the implementation of an AI system.
This can include:
— users who operate or interface with an AI system, as they can offer personal feedback on unwanted bias
that has affected them as individuals;
EXAMPLE 1 Front-line workers whose familiarity with digital technology is at a lower level than that of an AI
application’s main beneficiaries or target audience.
— decision-makers within an organization designing, developing, deploying, or using an AI system;
— AI partners (including AI auditors) who are required to perform conformity assessment on an AI system;
EXAMPLE 2 Auditing teams consider the diversity of the audited organization and the audited organization’s
target audience, customers and other interested parties when conducting the audit to prevent unwanted biases
affecting the audit.
— regulators including bodies that are required to review conformity assessment results in relation to an
AI system.
EXAMPLE 3 Regulatory staff consider the diversity of the auditors, the audited organization and the audited
organization’s target audience, customers and other interested parties when conducting the conformity
assessment to prevent unwanted biases affecting the review.
Organizations responsible for the deployment or operation of an AI system shall consider:
— AI subjects about whom automated decisions are made, or who share an operating environment with an
AI system;
EXAMPLE 4 A recommender system receiving input from a user ensures that variations in ability are
accounted for in applications that are used by native and non-native language speakers.
— recipients of information derived from an AI system who are not direct users (e.g. public authorities who
will make decisions based on AI-derived information curated by staff);
EXAMPLE 5 Analytical systems that analyse socioeconomic datasets to inform policy development can
propagate unwanted bias based on historic data that does not reflect current conditions.
— data subjects who do not directly interact with an AI system but whose data are used in training.
EXAMPLE 6 An online chess playing game uses data from real-world matches and tournaments. The players of
those matches are interested parties in this context.
5.1.2 Stakeholder needs and requirements definition
An organization shall identify and document requirements to reduce unwanted bias within an AI system as
well as in recommendations, decisions or other output generated by an AI system.
Considered sources of bias-related requirements can include:
— applicable legal requirements;
— customer expectations;
— internal goals, strategies and policies (e.g. an ethics policy);
— organizational processes and decisions as part of AI governance;
— surveys of past known failure modes, based on resources such as the organization’s documentation on
its prior system failures or in AI incident databases;

© ISO/IEC 2024 – All rights reserved
ISO/IEC TS 12791:2024(en)
— assumptions, processes, decisions and related activities made by individuals or groups across the AI
life cycle.
NOTE 1 Treatment leading to unwanted bias can include any kind of action or inaction, including perception, etc.
NOTE 2 Annex B lists particular examples of how unwanted bias can affect specific types of AI users.
ISO/IEC 22989:2022, 5.19 describes several AI stakeholder roles and sub-roles (e.g. AI provider, AI user, AI partner, AI
subject). AI stakeholders have different roles and responsibilities in treating unwanted bias throughout the life cycle.
An organization shall define and document intended operating conditions under which an AI system is to
be evaluated for bias. These can include the relevant groups of stakeholders or users and the geographical,
linguistic, socioeconomic or cultural context of deployment.
EXAMPLE For intended operating conditions of a speech recognition system it can be defined and documented
that both native and non-native speakers are expected.
The level of definition and documentation should be commensurate with the role of the organization within
the AI life cycle. For example, an organization producing a pre-trained ML model should anticipate a broad
number of intended operating conditions, while an organization conducting a live deployment should more
precisely specify the intended operating conditions.
Procedures, expectations and accountability mechanisms should be in place for relevant actors within the
organization to make use of the documentation they receive from others inside or outside the organization.
5.1.3 Procurement
AI producers or AI partners shall make information available regarding systems aspects that can affect
unwanted bias, subject to legal requirements. The exact extent of a non-disclosure by AI producers and
partners and the existence of legal requirements affecting disclosure shall be disclosed and justified.
Examples of such system aspects that can affect unwanted bias include:
— algorithm selection;
— hyperparameter tuning;
— model bias;
— bias in data sources.
Details are provided in ISO/IEC TR 24027. The conveyed information can include representative testing
results of the system with regards to unwanted bias.
Organizations should ensure that agreements with third parties include appropriate measures to treat the
risk of unwanted bias considering the role of the organization and third parties, in particular, where an
organization is unable to obtain full transparency on technical aspects of the system. Such measures shall
be documented and justified including divergence from the provisions of this document.
Information made available by data providers in the course of a procurement shall include, subject to legal
requirements:
— data provenance (including for training, validation and testing data);
EXAMPLE 1 This information can enable investigation of biases resulting from properties of the data source.
— data quality management policy and data quality check assessment results (including inter-annotator
agreement measurements);
EXAMPLE 2 This information can enable acceptance of the presence of an observed but mitigated bias that is
deemed appropriate for a given use case.
— data quality model and processing aspects (e.g. labelling processes used, types of machine learning
models or algorithms used).
© ISO/IEC 2024 – All rights reserved
ISO/IEC TS 12791:2024(en)
EXAMPLE 3 This information can be used to uncover biases associated with mechanisms that only the data
provider has visibility on, as they are part of their internal process.
The exact extent of a non-disclosure and the existence of legal requirements affecting disclosure by data
providers shall be disclosed and justified.
Information made available by data providers in the course of a procurement should include:
— the method of data collection;
— information on the working conditions for data labelling workers that can affect their human cognitive
biases and hence cause potential unwanted bias in the resulting dataset;
— the geographic locations in which the data labelling was undertaken;
— salient aggregated demographics of the data labelling workforce;
— data dictionaries and associated metadata to enable unwanted bias risk management.
Data providers should ensure that privacy of individuals and groups in the workforce is maintained.
Information made available by AI technology providers in the course of a procurement should include:
— intended context(s) of use and related assumptions;
— known system limitations;
— recommended patterns of interaction between humans and the AI system during use;
— relevant trade-offs in algorithms, machine learning algorithms and ML models development that can
affect or relate to unwanted bias;
EXAMPLE 4 How the model is used at inference time, where taking the argmax (argument of the maximum)
for classification problems, or using deterministic ranking, risks amplifying small biases in model scores;
— data collection, modification and curation processes that can relate to bias such as imputation or
augmentation;
— testing strategies used during the design and development or the verification and validation stage of the
AI system (including acceptance criteria and the use of proxies in ML modelling).
Information can also be made available by AI technology providers regarding the geographical or cultural
context of the design and development phase, as it can affect unwanted bias when this context differs
significantly from the context of deployment.
Implementation guidance: ML tools can be procured, developed in-house or a combination of the two.
Visibility into the kind of techniques used during algorithm, machine learning algorithm or ML model
development is important for the effective treatment of unwanted bias.
Information made available by data providers can enable the investigation of biases resulting from the
properties of the data source. This can enable acceptance of the presence of an observed but mitigated bias
that is deemed appropriate for a given use case or uncover biases associated with mechanisms used by the
data provider.
5.1.4 Data sources
Organizations shall document and evaluate unwanted bias in relation to:
— sources of data used by the AI system;
— data selection criteria and processes;

© ISO/IEC 2024 – All rights reserved
ISO/IEC TS 12791:2024(en)
— data collection procedures, including the:
— mechanisms for requesting informed consent and for revoking consent for future users;
— provenance of the data;
— collection, input, preprocessing, labelling and label cleaning mechanism;
— impacts of dataset collection on data subjects.
Documentation of data sources enables organizations to qualitatively identify potential biases and to
prioritize quantitative assessment in relation to those aspects. Quantitative assessment of unwanted bias is
not usually tractable and benefits from source information.
Test data used to assess performance shall be representative. Training data shall be representative when
using it to train an ML algorithm that is not specifically designed for leveraging non-representative data.
Using multiple data sources is one way to improve representativeness across diverse groups; when done so,
organizations shall determine whether the combination of datasets would introduce additional risks in relation
to data bias and, if so, the newly introduced biases shall be assessed and treated in accordance with 5.1.5.
Representativeness can be a function of data size, type, dimensionality and complexity. Imputations,
exclusions and augmentations in the dataset can affect representativeness. For ML models that cannot be
trained on data with missing values, the distribution of imputations, exclusions and augmentations applied
to make the dataset usable can affect representativeness.
For supervised ML, the distribution of the label values in training data and across at-risk groups is relevant to
representativeness. For example, in the case of binary supervised classifiers, the balance of the data among
the relevant groups can be evaluated by considering the ratio of positive and negative training examples
within each group.
Where appropriate for the use case, the following aspects of each data source shall be evaluated and
documented:
— intended use and purpose of the dataset created, including specific tasks;
— identification of dataset creators and sources of funding if applicable;
— composition of the dataset, including nature, size, labels, relations, errors, r
...