ISO/IEC 25040:2024

International
Standard
ISO/IEC 25040
Second edition
Systems and software
2024-09
engineering — Systems and
software Quality Requirements
and Evaluation (SQuaRE) — Quality
evaluation framework
Reference number
© ISO/IEC 2024
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on
the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below
or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
© ISO/IEC 2024 – All rights reserved
ii
Contents Page
Foreword .iv
Introduction .v
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
4 Concepts of quality evaluation . 2
4.1 Quality evaluation definition .2
4.2 Quality model and quality measures for quality evaluation .2
4.3 Measurement source .4
4.4 Tasks of quality evaluation .4
4.5 Quality rating module .4
4.6 Assessment using evaluation . .5
5 Quality evaluation process reference model . 5
5.1 Overview .5
5.2 Define the evaluation .7
5.2.1 Purpose .7
5.2.2 Outcomes .7
5.2.3 Activities .7
5.3 Design the evaluation .10
5.3.1 Purpose .10
5.3.2 Outcomes .10
5.3.3 Activities .11
5.4 Plan the evaluation . 12
5.4.1 Purpose . 12
5.4.2 Outcomes . 12
5.4.3 Activities . 13
5.5 Execute the evaluation .14
5.5.1 Purpose .14
5.5.2 Outcomes .14
5.5.3 Activities .14
5.6 Conclude the evaluation .14
5.6.1 Purpose .14
5.6.2 Outcomes .14
5.6.3 Activities . 15
Annex A (informative) Measurement source . 17
Annex B (informative) Examples of four types of quality evaluations .20
Annex C (informative) Quality evaluation process implementation through system/software
life cycle processes .24
Bibliography .29

© ISO/IEC 2024 – All rights reserved
iii
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical activity.
ISO and IEC technical committees collaborate in fields of mutual interest. Other international organizations,
governmental and non-governmental, in liaison with ISO and IEC, also take part in the work.
The procedures used to develop this document and those intended for its further maintenance are described
in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for the different types
of document should be noted. This document was drafted in accordance with the editorial rules of the ISO/
IEC Directives, Part 2 (see www.iso.org/directives or www.iec.ch/members_experts/refdocs).
ISO and IEC draw attention to the possibility that the implementation of this document may involve the
use of (a) patent(s). ISO and IEC take no position concerning the evidence, validity or applicability of any
claimed patent rights in respect thereof. As of the date of publication of this document, ISO and IEC had not
received notice of (a) patent(s) which may be required to implement this document. However, implementers
are cautioned that this may not represent the latest information, which may be obtained from the patent
database available at www.iso.org/patents and https://patents.iec.ch. ISO and IEC shall not be held
responsible for identifying any or all such patent rights.
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and expressions
related to conformity assessment, as well as information about ISO's adherence to the World Trade
Organization (WTO) principles in the Technical Barriers to Trade (TBT) see www.iso.org/iso/foreword.html.
In the IEC, see www.iec.ch/understanding-standards.
This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 7, Software and systems engineering.
This second edition cancels and replaces the first edition (ISO/IEC 25040:2011), which has been technically
revised.
The main changes are as follows:
— alignment with the other SQuaRE divisions: quality management, model, measurement, and requirements;
— alignment with other standards for system/software life cycle processes and requirements engineering
processes;
— expansion of its target entities from software to ICT products, data, and IT services;
— expansion of types of quality evaluation from only requirements conformity to four types: quality
evaluation for suitability to a specific use, for qualification to quality standard, for conformity checking
to requirements, and for suitability to the market;
— clarification of concepts relating to quality evaluation;
— provision of more practical guidelines for planning a quality evaluation.
Any feedback or questions on this document should be directed to the user’s national standards
body. A complete listing of these bodies can be found at www.iso.org/members.html and
www.iec.ch/national-committees.

© ISO/IEC 2024 – All rights reserved
iv
Introduction
Many systems and services are now deeply embedded into social infrastructures used in daily life. This
requires them to achieve much higher quality; for example, connected systems need to be interoperable,
secure, reliable, maintainable, and usable. Therefore, quality evaluation becomes ever more important.
The result of quality evaluation is used to objectively judge the value of the target entity in various business
situations, including:
— outsourcers judging whether the target entity satisfies their quality requirements, in the case of
outsourcing it;
— consumers or acquirers determining which product or service to be selected for their application, in the
case of acquisition.
This document provides requirements and recommendations for quality evaluation, as well as guidance for
its tasks.
Target entities for quality evaluation include ICT (information and communication technology) products
(systems, software products, etc.), data, and IT services. The quality model defined by ISO/IEC 2501n
provides comprehensive quality characteristics and subcharacteristics according to the types of target
entities. ISO/IEC 2502n defines quality measures corresponding to those quality models.
This document is one of the standards on SQuaRE developed by ISO/IEC JTC 1/SC 7 (ISO/IEC 25000 to
ISO/IEC 25099). Figure 1 (adapted from ISO/IEC 25000) illustrates the organization of the standards on
SQuaRE developed by ISO/IEC JTC 1/SC 7. Similar standards are grouped into divisions. Each division
provides guidance and resources for performing a different function in ensuring system and software
product quality.
— ISO/IEC 2500n - quality management division. The International Standards that form this division define
all common models, terms, and definitions referred to by all other International Standards on SQuaRE
developed by ISO/IEC JTC 1/SC 7. This division also provides requirements and guidance for a supporting
function that is responsible for the management of the requirements, specification, and evaluation of
software product quality. Practical guidance on the use of the quality models is also provided.
— ISO/IEC 2501n - quality model division. The International Standards that form this division present
detailed quality models for computer systems and software products, data, IT services and quality-in-use.
— ISO/IEC 2502n - quality measurement division. The International Standards that form this division
include a quality measurement framework, mathematical definitions of quality measures, and practical
guidance for their application. Examples are given of quality measures for internal and external property
of product, data, IT services and quality-in-use. Quality measure elements (QME) forming foundations
for quality measures for internal and external property of product are defined and presented.
— ISO/IEC 2503n - quality requirements division. The International Standards that form this division help
specify quality requirements based on quality models and quality measures. These quality requirements
can be used in the process of eliciting quality requirements for information systems and IT services to be
developed or as input for an evaluation process.
— ISO/IEC 2504n - quality evaluation division. The International Standards that form this division provide
requirements, recommendations and guidelines for quality evaluation for information systems and IT
services.
— ISO/IEC 25050 to ISO/IEC 25099 - SQuaRE extension division. These International Standards currently
include requirements for quality of ready-to-use software product (RUSP), common industry formats
for usability reports, and quality models and measures for new technologies such as cloud services and
artificial intelligence.
© ISO/IEC 2024 – All rights reserved
v
Figure 1 — Organization of standards on SQuaRE developed by ISO/IEC JTC 1/SC 7
The SQuaRE standards can be used in conjunction with ISO/IEC/IEEE 12207 and ISO/IEC/IEEE 15288,
particularly the processes for the specification and evaluation of quality requirements. ISO/IEC 25030
describes how quality models and measures can be used for systems and software quality requirements,
and ISO/IEC 25040 describes how the quality models and measures can be used for systems and software
quality evaluation.
The SQuaRE standards can also be used in conjunction with standards on software process assessment
developed by ISO/IEC JTC 1/SC 7, which provide:
— a framework for software product quality definition in the customer-supplier process;
— support for quality review, verification, and validation, as well as a framework for establishing
quantitative quality characteristics;
— support for setting organizational quality goals in the management process.
The SQuaRE standards can be used in conjunction with ISO 9001 (which is concerned with quality
management system) to provide:
— support for setting quality goals;
— support for design review, verification, and validation.

© ISO/IEC 2024 – All rights reserved
vi
International Standard ISO/IEC 25040:2024(en)
Systems and software engineering — Systems and software
Quality Requirements and Evaluation (SQuaRE) — Quality
evaluation framework
1 Scope
This document provides the framework for quality evaluation of ICT (information and communication
technology) products (including software products), data, and IT services, which includes its concepts, and
requirements, and recommendations for its processes to be implemented and enhanced.
This document is applicable for those who need to perform quality evaluations on target entities, including
development organization (integrators, developers, and quality assurance group), acquirers, IT service
providers, and independent evaluators.
This document does not provide specific test methods for quality evaluation or guidance on utilizing
evaluation results.
2 Normative references
The following documents are referred to in the text in such a way that some or all of their content constitutes
requirements of this document. For dated references, only the edition cited applies. For undated references,
the latest edition of the referenced document (including any amendments) applies.
ISO/IEC 25000, Systems and software engineering — Systems and software Quality Requirements and
Evaluation (SQuaRE) — Guide to SQuaRE
ISO/IEC 25030, Systems and software engineering — Systems and software quality requirements and evaluation
(SQuaRE) — Quality requirements framework
3 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO/IEC 25000 and the following apply.
ISO and IEC maintain terminology databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https:// www .iso .org/ obp
— IEC Electropedia: available at https:// www .electropedia .org/
3.1
assessment
action of comprehensively evaluating the target entity based on documented criteria for a specific purpose
Note 1 to entry: Such purposes can include determining acceptance or release of the target entity.
3.2
implementation rating module
quality rating module (3.8) that can be directly applied to target entity

© ISO/IEC 2024 – All rights reserved
3.3
information need
insight necessary to manage objectives, goals, risks and problems
[SOURCE: ISO/IEC/IEEE 15939:2017, 3.12]
3.4
integrity level
degree of confidence that the system-of-interest meets the associated integrity level claim
[SOURCE: ISO/IEC/IEEE 15026-3:2023, 3.1, modified — Notes to entry have been removed.]
3.5
measurement source
set of artefacts used for quality measures when performing a quality evaluation (3.7)
3.6
quality analysis
analysis of rating results for multiple quality properties to determine the objective score or acceptability for
the quality of the target entity
3.7
quality evaluation
systematic examination of the extent to which an entity is capable of fulfilling specified requirements
[SOURCE: ISO/IEC/IEEE 24765:2017, 3.3267, modified — Note 1 to entry has been removed.]
3.8
quality rating module
set of quality measures, operational environment, and methods for conducting quality measurements and
quality ratings on a specific category of target entities
3.9
template rating module
quality rating module (3.8) in which the measurement environment and rating method are parameterized
for future use
4 Concepts of quality evaluation
4.1 Quality evaluation definition
A quality evaluation involves performing a series of tasks to generate objective and quantitative data
regarding the quality of the target entity. This is achieved by measuring, rating, and interpreting the quality
attributes of the target entities using established quality models and measures. The ultimate objective of
a quality evaluation is to offer crucial and unbiased information that supports comprehensive decision-
making during the planning, development, delivery, acceptance, maintenance, and procurement stages of
the target entities.
4.2 Quality model and quality measures for quality evaluation
The SQuaRE standards offer quality models and measures, as presented in Table 1, specifically designed for
these three types of target entities which are ICT products, data, and IT services.

© ISO/IEC 2024 – All rights reserved
Table 1 — Target entity types and their associated quality models and measures
Quality to be
Target entity type Quality model Quality measures
evaluated
IT service / ISO/IEC 25019 ISO/IEC 25022
Quality-in-use
ICT product Quality-in-use model Measurement of quality-in-use
IT service ISO/IEC TS 25011 ISO/IEC TS 25025
IT service
quality IT service quality models Measurement of IT service quality
ISO/IEC 25023
ISO/IEC 25010
ICT product Product quality Measurement of system and soft-
Product quality model
ware product quality
ISO/IEC 25012 ISO/IEC 25024
Data Data quality
Data quality model Measurement of data quality
The quality model presents a framework that outlines the essential quality characteristics that the target
entity should possess. Quality measurement serves as a method for quantifying these quality characteristics
based on the established quality model. In the case of an ICT product or IT service as the target entity, the
quality-in-use model can be utilized to evaluate the impact of the target entity during usage.
NOTE 1 When evaluating ICT products (or IT services), the quality-in-use model or the product quality model (or
IT service quality model) can be selected depending on the purpose of the quality evaluation. When evaluating data
quality, the data quality model is the only option.
When evaluating a particular target entity, the important quality (sub)characteristics vary by its category.
NOTE 2 ISO/IEC TR 12182 provides a framework for categorizations of IT systems and software, and a guide for its
application.
EXAMPLE In the case of a server device as the target entity, usability is not a significant factor as there are
no direct user interactions involved. When evaluating the time behaviour of the server, the relevant information
requirement shifts from response time to the throughput of requests handled by the server.
Quality properties represent the information needs pertaining to the quality of the target entity. These
properties are measured and quantified using quality measures. It is worth noting that in certain cases,
multiple measures can correspond to a single quality property. Various measurement methods, including
testing, inspection, static analysis, and document survey, are utilized. See Figure 2.
Figure 2 — Relationship among quality characteristics, quality properties and quality measures

© ISO/IEC 2024 – All rights reserved
4.3 Measurement source
A measurement source is a set of artefacts used for quality measurement when conducting a quality
evaluation. The measurement source depends on what is available at the time of the evaluation. Design
artefacts (design specifications, prototypes, inspection-related documents, test specifications, etc.) are
available even when the target entities themselves do not yet exist. On the other hand, final products
(executable objects, source code, data, usage manuals, maintenance manuals, results of inspection and/or
testing, etc.) are available near their delivery. The measurement source determines the applicable quality
measures.
NOTE Annex A shows a list of artefacts that can be included in measurement sources, what can be measured from
them, and information on how to measure them.
4.4 Tasks of quality evaluation
Quality evaluation consists of the following three tasks (see Figure 3).
— Quality measurement: quantitatively measuring the quality property of the target entity using specific
quality measure. Its objective is to obtain objective and quantitative information about the quality
properties of the target entity.
— Quality rating: comparing the measured values obtained from quality measurement against
predetermined quality rating levels to determine the appropriate quality level for each quality property
of the target entity. For instance, the measured response time is classified as the level of "excellent" if it
falls within the range corresponding to the level.
— Quality analysis: analysing the quality rating results obtained from multiple quality properties. The
focus is on assessing the acceptability of the target entity, taking into account each individual quality
characteristic or subcharacteristic, as well as the overall quality. This analysis also includes identifying
strengths and weaknesses for each quality attribute and generating recommendations based on the
assessment findings.
Figure 3 — Tasks of quality evaluation
4.5 Quality rating module
A quality rating module is a cohesive collection of measures, operational environment, and rating methods
for performing quality measurements and ratings on a category of target entities. By utilizing the quality
rating module, one can obtain quality rating results that align with the specified information needs.
The following are elements that a quality rating module can encompass:
— quality (sub)characteristics to be evaluated;
— categories of target entities eligible for evaluation;

© ISO/IEC 2024 – All rights reserved
— quality measures employed for evaluation;
— measurement conditions and operational environment, including relevant parameters;
— defined quality rating levels;
— resource requirements;
— assumptions and constraints applicable to the quality rating module (e.g. this module does not need to
consider potential interaction problems arising from inexperienced or untrained users; this is because
the users interacting with the target entity are supposed to be trained adequately.)
An implementation rating module is a quality rating module that can be directly applied to existing target
entities. On the other hand, a template rating module is a parameterized quality rating module that allows for
flexibility in the measurement environment and rating method, making it suitable for future use. To create
an implementation rating module, the parameters of a template rating module are applied and customized.
NOTE The category of the target entity plays a significant role in determining the quality (sub)characteristics to
evaluate, their relative significance, the applicable quality rating modules (including rating level setting and range of
acceptance), and the pass/fail criteria for quality analysis. ISO/IEC TR 12182 offers an example of categorization for
ICT products that can be referenced in this context.
4.6 Assessment using evaluation
An assessment is a type of decision-making process that occurs during various stages, such as planning,
development, shipping, acceptance, maintenance, or purchase of the target entity. It involves evaluating the
quality of target entity based on specific criteria or requirements.
A quality evaluation can provide objective information about the quality of the target entities for assessment
purposes. The rigor of the quality evaluation largely depends on the type of assessment being conducted.
For example, in industries such as aviation or healthcare, safety assessments are critical, involving analysing
safety protocols, equipment performance, and adherence to regulatory standards. The rigor of the quality
evaluation for such assessment must be high to ensure the safety of individuals and prevent potential
hazards. On the other hands, in the early stages of product development or project planning, there is a case
that preliminary assessments are conducted to gather initial insights or make rough estimates as a starting
point for further evaluation and decision-making. In this case, the rigor of the quality evaluation can be low
because the data is not as accurate as in later stages.
Assessments can be involved with various activities such as identifying target entities for acquisition,
making acceptance decisions, determining delivery schedules, deciding on phase transitions, evaluating
market value at the end of each phase.
NOTE While the results of a quality evaluation can be used as input for certain assessments, quality evaluation
itself is conducted independently from any specific assessment.
5 Quality evaluation process reference model
5.1 Overview
The quality evaluation process reference model intends to present the general quality evaluation process. It
consists of the five steps:
a) define the evaluation;
b) design the evaluation;
c) plan the evaluation;
d) execute the evaluation;
© ISO/IEC 2024 – All rights reserved
e) conclude the evaluation.
Figure 4 illustrates the activities to be performed in each step and the expected outputs.
Figure 4 — Quality evaluation process reference model
When the organization that needs the evaluation proactively performs the "define evaluation" step, the
remaining steps can be outsourced. "Define the evaluation" and “Plan the evaluation” should be considered
as parallel steps.
NOTE 1 This evaluation process places “Plan the evaluation” in the third step, because only after the "Define the
evaluation" and "Design the evaluation," the tasks, schedule, and costs of the quality evaluation can be determined
accurately. However, since the "Define the evaluation " and "Design the evaluation " also require the allocation of
skilled resources, planning for them is necessary in the first step of the evaluation process. Therefore, it is appropriate
to think of “Plan the evaluation " as a parallel process with "Define the evaluation" and “Design the evaluation”.
NOTE 2 An example of quality evaluation process implementation through system/software life cycle processes is
shown in Annex C.
© ISO/IEC 2024 – All rights reserved
5.2 Define the evaluation
5.2.1 Purpose
This step aims to define the scope of the evaluation by establishing the purpose, evaluation criteria, target
entities, and other relevant factors from a business and organizational perspective.
5.2.2 Outcomes
As a result of the successful implementation of the step “Define the evaluation”.
a) Purpose of quality evaluation is defined, in order to elaborate quality evaluation activities and tasks to
meet the requirements to evaluation from stakeholders.
b) Quality evaluation criteria, needed for determination of quality of target entities through evaluation,
are defined.
c) Target entities needed for quality evaluation are identified, accessible and available.
d) Requirements for rigor of the evaluation are defined to be a basis of considering enable approaches of
the evaluation.
5.2.3 Activities
5.2.3.1 Establish the purpose
The purpose of the quality evaluation shall be defined. Generally, the quality evaluation is conducted with
the following objectives (four types of quality evaluation):
(T1) Quality evaluation for suitability to a specific context of use:
— Purpose: obtaining information for IT service/ ICT product/ data selection for a certain decision
making
— What to determine:
— whether the candidate entities meet some specific requirements;
— which entities of them are better overall;
— the strengths and weaknesses of each entity.
— Quality criteria used: a defined set of quality requirements
(T2) Quality evaluation for qualification to quality standard:
— Purpose: obtaining quality information to improve the quality of the target entity based on the
quality standards
— What to determine:
— whether the entity complies with established standards;
— its strengths and weaknesses of the target entity from a quality standpoint.
— Quality criteria used: the governmental/industrial standards
(T3) Quality evaluation for checking requirements satisfaction:
— Purpose: verifying the fulfilment of the agreed requirements

© ISO/IEC 2024 – All rights reserved
— What to determine:
— whether the target entity meets the quality requirements.
— Quality criteria used: the agreed requirements
(T4) Quality evaluation for suitability to the market
— Purpose: gathering information for management decisions
— What to determine:
— how well the target entity meets the needs of the target market.
— Quality criteria used: the requirements derived from the potential stakeholder needs (in some
cases, incrementally developed)
When different types of quality evaluations are mixed and conducted as a single evaluation, the requirements
for each of them shall be defined separately.
EXAMPLE The supplier can combine the T2 and T3 evaluations when it is not possible to establish the evaluation
criteria for the T3 evaluation solely based on the quality requirements agreed upon by both parties.
5.2.3.2 Identify target entities
All entities needed for the evaluation shall be identified or defined.
The user of the quality evaluation is either the acquirer or the supplier of the target entities. From the
viewpoints of the user, the target entities can be either ordered, commercially available, or existing.
NOTE An existing ICT product or IT service can also be a target entity (e.g. to determine whether an existing ICT
product or IT service needs to be replaced or not, based on the quality evaluation results for it).
EXAMPLE When an ICT product under development is in acceptant testing, the product itself will be the target
entity. When an ICT product is acquired among several candidates of commercially available products, the candidate
products will be the target entities. When data or software components are to be acquired, which are suitable to use in
the ICT product under development, these components can be the target entities.
5.2.3.3 Define quality evaluation criteria
The quality evaluation criteria shall be defined or identified.
Quality evaluation criteria are a set of specific quality requirements used to evaluate the quality of the
target entities. They can include factors such as functional suitability, reliability, performance efficiency,
compatibility, interaction capability, maintainability, flexibility, security, safety and compliance with
relevant standards or regulations. These criteria help in objectively evaluating and determining the level of
quality achieved by the entity.
A quality requirement shall have the following items in accordance with ISO/IEC 25030:
— target entity;
— selected characteristic;
— user and task (only for quality-in-use requirements);
— quality goal with conditions;
— quality measure;
— target value;
— acceptable range of values.
© ISO/IEC 2024 – All rights reserved
The above items can be used as quality evaluation criteria.
Sometimes given requirements specifications do not provide a sufficient set of quality requirements for
quality evaluations or some quality requirements in them lack the above items. It is required to supplement
them because they are necessary to choose the right skills, tools, and techniques for quality evaluation in
the evaluation design and planning.
In particular, the following factors should be considered to establish quality evaluation criteria.
a) Criticality and integrity level
Quality evaluation criteria can vary depending on the context and nature of the entity being evaluated.
When the evaluation is related to a system integrity level, the degree required for the quality characteristics
and subcharacteristics of the target entity must fulfil the criticality of its expected use.
NOTE 1 ISO/IEC/IEEE 15026-3 defines system and software integrity levels. The required integrity level of the
software largely determines the rigor and formality of the evaluation.
EXAMPLE 1 Evaluation techniques to be applied to the functionality characteristic according to different evaluation
levels requirements, from less demanding levels to more demanding levels includes:
— functional or black box testing;
— inspection of development documentation guided by checklists;
— unit testing with test coverage criteria.
b) Source of quality evaluation criteria
Quality evaluation criteria can be established using predefined sources. There are two types of sources of
quality evaluation criteria:
1) diagnostic model, which is predefined quality evaluation criteria for a specific category of target entities
in a specific domain, including a set of prioritized quality requirements sufficient for evaluating the
entities;
2) requirements specification, which consists of a set of quality requirements for a specific target entity to
develop or acquire.
EXAMPLE 2 Diagnostic models can include government-established standards, industry group standards, and
commonly used standards within specific organizations or projects.
EXAMPLE 3 Requirements specification can include mutually agreed-upon requirements specifications between
the acquirer and supplier, and dedicated requirements specifications developed specifically for the evaluation process.
NOTE 2 Quality evaluations using requirements specifications are not suitable for scoring the quality of the entity
since they can only indicate whether the entity to be evaluated meets the requirements. In case the development
organization has some diagnostic model for a specific domain, it is possible to conduct the quality evaluation using
a combination of the requirements specification and the diagnostic model for a more objective and comprehensive
quality evaluation.
The evaluation type (except T4) determines available sources of quality evaluation criteria, as outlined in
Table 2. For T4, both diagnostic models and requirement specifications can be available.
When establishing quality evaluation criteria, the following aspects should be clarified:
— source for the quality evaluation criteria, including how they are obtained and derived;
— the expected quality outlined in the source, including reasons for its importance and what needs to be
demonstrated.
© ISO/IEC 2024 – All rights reserved
Table 2 — Type of quality evaluation and available source of quality evaluation criteria
Available source of quality evaluation
Type of quality evaluation
criteria
Quality evaluation for suitability to a
T1 Requirements specification
specific use
Quality evaluation for qualification to
T2 Diagnostic model
quality standard
Quality evaluation for checking require-
T3 Requirements specification
ments satisfaction
Quality evaluation for suitability to the Requirements specification / Diagnos-
T4
market tic model
NOTE 3 Annex B shows how to determine quality criteria for several business contexts.
5.2.3.4 Define requirements for the rigor of evaluation
The requirements for the rigor of the quality evaluation shall be defined. The term "rigor" in evaluation
refers to the level of thoroughness, precision, and strictness applied during the evaluation process to
ensure the accuracy, reliability (e.g., repeatability and reproducibility of measurement), and validity of the
measurement and evaluation results.
NOTE 1 Further information on reliability of measurement can be found in ISO/IEC 25020:2019, Annex B.
The rigor of the evaluation becomes crucial when the evaluation results are directly utilized in judgments,
such as determining whether the contract conditions have been met. Especially when the evaluation results
are employed as objective data for making significant managerial decisions, the rigor of the evaluation
should be carefully considered.
In order to establish these requirements, the following factors should be taken into consideration:
a) coverage of information needs;
b) objectivity and accuracy of measured values (how strictly the measurements need to be conducted);
c) acceptability of the rating levels;
d) acceptability of weighting for scoring;
e) transparency of the quality evaluation.
The extent of factors a) and b) significantly influences the evaluation cost.
NOTE 2 To define the rigor of the evaluation, Annex B lists considerations for each type of quality evaluation.
5.3 Design the evaluation
5.3.1 Purpose
The purpose of this process is to design the quality evaluation, including its target entities and their
components that need to undergo individual evaluation, methods to be applied for quality measurement,
rating, and analysis, and its output.
5.3.2 Outcomes
As a result of the successful implementation of the step “Design the evaluation”.
a) Components needed for quality evaluation are identified.
b) Applicable quality rating modules are selected for measuring and rating quality of target entities.

© ISO/IEC 2024 – All rights reserved
c) Applicable quality analysis methods are selected to analyse quality from the results of quality rating.
d) Outputs of quality evaluation, needed for intended use of stakeholders, are defined.
5.3.3 Activities
5.3.3.1 Identify components to be evaluated
In quality evaluation, the entities and components that need to undergo individual evaluation shall be
identified. These evaluations are then integrated to produce a comprehensive quality evaluation result
through quality analysis.
When evaluating an IT service, the components can be various ICT products and data. ICT products can also
have their own set of components, including other ICT products and data.
When an ICT product consists of components belonging to different system categories, certain rating
modules can only be applied to specific categories of components based on the specified information needs
(quality properties).
EXAMPLE In case of an ICT product comprising terminal devices with user interfaces and server devices
responsible for handling a large volume of processing requests, two components can be identified, and it is important
to determine the appropriate rating modules to be applied to each component.
5.3.3.2 Select and implement quality rating modules
Quality rating modules for the target entities (or their components) shall be selected and implemented.
The following procedures should be used to prepare a quality rating module.
a) Select quality (sub)characteristics important for the target entities.
b) Specify information needs on each of the (sub)characteristics, i.e., what are wanted to know about
quality of the target entities from the views of the sel
...