ISO/IEC 20741:2017

INTERNATIONAL ISO/IEC
STANDARD 20741
First edition
2017-05
Systems and software engineering —
Guideline for the evaluation and
selection of software engineering tools
Ingénierie des systèmes et du logiciel — Lignes directrices pour
l’évaluation et le choix des outils d’ingénierie logicielle
Reference number
ISO/IEC 20741:2017(E)
©
ISO/IEC 2017

---------------------- Page: 1 ----------------------
ISO/IEC 20741:2017(E)

COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2017, Published in Switzerland
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized otherwise in any form
or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior
written permission. Permission can be requested from either ISO at the address below or ISO’s member body in the country of
the requester.
ISO copyright office
Ch. de Blandonnet 8 • CP 401
CH-1214 Vernier, Geneva, Switzerland
Tel. +41 22 749 01 11
Fax +41 22 749 09 47
copyright@iso.org
www.iso.org
ii © ISO/IEC 2017 – All rights reserved

---------------------- Page: 2 ----------------------
ISO/IEC 20741:2017(E)

Contents Page
Foreword .v
Introduction .vi
1 Scope . 1
2 Normative references . 2
3 Terms and definitions . 2
4 Abbreviated terms . 3
5 Overview of evaluation and selection of software engineering tools .3
5.1 Introduction of the evaluation and selection of software engineering tools. 3
5.2 Framework of the evaluation and selection of software engineering tools . 4
5.3 General process considerations . 4
5.3.1 Sequencing of processes . 4
5.3.2 Reducing cost and risk . 5
6 Preparation process . 5
6.1 Purpose . 5
6.2 Outcomes . 6
6.3 Activities and tasks . 6
6.3.1 Goal setting . 6
6.3.2 Establishing selection criteria . 7
6.3.3 Project planning and control . 7
7 Structuring process . 8
7.1 Purpose . 8
7.2 Outcomes . 8
7.3 Activities and tasks . 9
7.3.1 Requirements definition . 9
7.3.2 Software engineering tool information gathering .10
7.3.3 Identifying final candidate software engineering tools .11
8 Evaluation process .11
8.1 Purpose .11
8.2 Outcomes .12
8.3 Activities and tasks .12
8.3.1 Preparing for evaluation .12
8.3.2 Evaluating software engineering tools .13
8.3.3 Evaluation reporting . .14
9 Software engineering tool selection process .14
9.1 Purpose .14
9.2 Outcomes .15
9.3 Activities and tasks .15
9.3.1 Preparing for selection .15
9.3.2 Applying the selection algorithm .15
9.3.3 Recommending a selection decision .15
9.3.4 Validating the selection decision .15
10 General software tool characteristics .16
10.1 Overview .16
10.2 Characteristics related to software engineering tool usage functionality .16
10.2.1 Overview .16
10.2.2 Software engineering tool operation environment characteristics .16
10.2.3 Software engineering tool integrability characteristics .17
10.2.4 Software engineering tool application characteristics .18
10.3 General quality characteristics.20
10.3.1 Overview .20
© ISO/IEC 2017 – All rights reserved iii

---------------------- Page: 3 ----------------------
ISO/IEC 20741:2017(E)

10.3.2 Functional suitability characteristics .20
10.3.3 Performance efficiency characteristics .20
10.3.4 Compatibility characteristics .21
10.3.5 Usability characteristics .21
10.3.6 Reliability characteristics .22
10.3.7 Security characteristics .23
10.3.8 Maintainability characteristics .24
10.3.9 Portability characteristics .25
10.4 General characteristics not related to quality .26
10.4.1 Overview .26
10.4.2 Acquisition process characteristics .26
10.4.3 Implementation characteristics .27
10.4.4 Support indicators characteristics .27
10.4.5 Evaluation or certification characteristics .28
Annex A (informative) Examples of selection algorithms .29
Annex B (informative) Evaluation report contents .32
Bibliography .34
iv © ISO/IEC 2017 – All rights reserved

---------------------- Page: 4 ----------------------
ISO/IEC 20741:2017(E)

Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical
activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international
organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the
work. In the field of information technology, ISO and IEC have established a joint technical committee,
ISO/IEC JTC 1.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for
the different types of documents should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www .iso .org/ directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent
rights. Details of any patent rights identified during the development of the document will be in the
Introduction and/or on the ISO list of patent declarations received (see www .iso .org/ patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation on the voluntary nature of standards, the meaning of ISO specific terms and
expressions related to conformity assessment, as well as information about ISO’s adherence to the
World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see the following
URL: w w w . i s o .org/ iso/ foreword .html.
This document was prepared by Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 7, Software and systems engineering.
© ISO/IEC 2017 – All rights reserved v

---------------------- Page: 5 ----------------------
ISO/IEC 20741:2017(E)

Introduction
Within systems and software engineering, software engineering tools represent a major part of the
supporting technologies used to develop and maintain information technology systems. Their selection
is carried out with careful consideration of both the technical and management requirements.
The objective of an evaluation process is to provide quantitative and comparable results of all candidate
alternatives. The final selection can then be based on these results. To be widely useful and accepted,
the software engineering tool evaluation and selection processes are supposed to help both the users
and the suppliers of software engineering tools. The more objective, repeatable, and impartial the
evaluation and selection processes are, the more widely acceptable they are. The information and
guidance outlined in this document are intended to lead to more cost-effective selections of software
engineering tools and to a greater uniformity in how software engineering tool functions and features
are described.
For evaluating and selecting software engineering tools, a set of processes providing a procedure for
evaluation and selection, a list of capabilities providing scope of functional requirements, and a list of
characteristics providing scope of non-functional requirements are needed.
Evaluation and selection of software engineering tools is usually performed within a specific, purpose-
oriented tool area for practical reasons, to manage the scope of evaluation and selection. Examples
of such tool areas are requirements engineering tools and configuration management tools. Lists of
capabilities are tool area specific, but the list of characteristics and the set of evaluation and selection
processes are more generic for all software engineering tool areas.
This document defines a set of processes and a list of characteristics which can be used by all software
engineering tool areas. This document can be used together with any tool area-specific standard which
defines list of capabilities for the tool area.
International standards defining lists of capabilities for specific tool areas have been published, such
as ISO/IEC 30130 for “software testing tools”, ISO/IEC TR 24766 for “requirements engineering tools”,
and ISO/IEC TR 18018 for “configuration management tools”. Lists of capabilities for other tool areas of
software engineering can be developed as a series of standards according to their priority.
It is supposed in this document that tool area is decided before starting the evaluation and selection.
It is recommended that the decision would be based on ISO/IEC 15940 which defines the software
engineering service for each tool area.
This document adopts the general model of software product quality characteristics and sub-
characteristics defined in ISO/IEC 25010 and gives additional guidance how to apply the model when
the software product is a software engineering tool. The document follows also the software product
evaluation model defined in ISO/IEC 25041.
vi © ISO/IEC 2017 – All rights reserved

---------------------- Page: 6 ----------------------
INTERNATIONAL STANDARD ISO/IEC 20741:2017(E)
Systems and software engineering — Guideline for the
evaluation and selection of software engineering tools
1 Scope
This document gives guidelines for the evaluation and selection of software engineering tools, covering
a partial or full portion of the software engineering life cycle.
It establishes processes and activities to be applied for the evaluation of software engineering tools and
selecting the most appropriate software engineering tools from several candidates.
It establishes, for selected processes, the tasks and activities that can be applied for the evaluation of
software engineering tools and selecting the most appropriate software engineering tools from several
candidates.
It establishes processes that can be applied for the evaluation of software engineering tools and
selecting the most appropriate software engineering tools from several candidates.
As these processes are generic, organizations can adapt these generic processes to meet organizational
needs. The software engineering tool evaluation and selection processes can be viewed in the larger
context of the organization’s technology adoption process.
This document provides the following:
a) guidance on identifying organizational requirements for software engineering tools;
b) guidance on mapping those requirements to software engineering tool characteristics to be
evaluated;
c) a process for selecting the most appropriate software engineering tool from several tools, based on
measurements of the defined characteristics.
NOTE 1 Guidance on mapping those requirements to software engineering tool capabilities to be evaluated is
not covered by this document, but is covered by a series of standards for each tool area.
Primary users of this document are organizations that intend to adopt software engineering tools
to support their software life cycle processes. Software tool suppliers can also use this document to
describe characteristics of their software engineering tools.
This document is not intended to apply to:
a) software engineering frameworks whose purpose is to provide mechanisms for data, control and
presentation integration;
b) general purpose tools (e.g. word processors, spreadsheets) which can be used in software
engineering activities, nor software engineering tools of very narrow scope or specific purpose
(e.g. a compiler);
c) planning for the implementation of software engineering tools within an organization.
NOTE 2 A user of this document can make the best possible selection of a software engineering tool and yet
have no guarantee of a successful implementation.
The methods described in this document are useful not only for the selection of software engineering
tools, but for any project where COTS/FOSS software can be selected instead of engaging in new
software development.
© ISO/IEC 2017 – All rights reserved 1

---------------------- Page: 7 ----------------------
ISO/IEC 20741:2017(E)

To follow the guidance provided in this document consists in applying the activities and tasks that are
attached to the defined processes to evaluate and select software. Organizations using this document
for trade purposes can specify the minimum set of processes and their related activities and tasks,
suitable to their given application.
2 Normative references
There are no normative references in this document.
3 Terms and definitions
For the purposes of this document, the following terms and definitions apply.
ISO and IEC maintain terminological databases for use in standardization at the following addresses:
— IEC Electropedia: available at http:// www .electropedia .org/
— ISO Online browsing platform: available at http:// www .iso .org/ obp
3.1
atomic sub-characteristic
lowest level sub-characteristics
Note 1 to entry: The highest level evaluation categories are called characteristics. Characteristics are usually
subdivided into sub-characteristics. At the lowest level, when no further subdivision is appropriate, the sub-
characteristics are referred to as atomic sub-characteristics.
3.2
characteristic
aspect of a product by which it can be described and evaluated
Note 1 to entry: A characteristic can be refined into multiple levels of sub-characteristics that bear on its ability
to satisfy stated or implied needs.
3.3
measure (noun)
variable to which a value is assigned as the result of measurement
Note 1 to entry: The term “measures” is used to refer collectively to base measures, derived measures, and
indicators.
[SOURCE: ISO/IEC 15939:2007, 2.15, modified — The words “plural form” have been changed to “term”.]
3.4
measure (verb)
make a measurement
[SOURCE: ISO/IEC 25040:2011, 4.39]
3.5
measurement
set of operations having the object of determining a value of a measure
Note 1 to entry: Measurement can include assigning a qualitative category such as the language of a source
program (ADA, C, COBOL, etc.).
[SOURCE: ISO/IEC 15939:2007, 2.17, modified — Note 1 to entry has been changed.]
3.6
software engineering tool
software product that assists software engineers by providing automated support
2 © ISO/IEC 2017 – All rights reserved

---------------------- Page: 8 ----------------------
ISO/IEC 20741:2017(E)

3.7
rating
action of mapping the measured value to the appropriate rating level
Note 1 to entry: Used to determine the rating level associated with the software for a specific quality
characteristic.
Note 2 to entry: Rating and rating levels can be applied to characteristics other than quality characteristics.
3.8
rating level
scale point on an ordinal scale which is used to categorize a measurement scale
Note 1 to entry: The rating level enables software to be classified (rated) in accordance with the stated or implied
needs (see 8.2).
Note 2 to entry: Appropriate rating levels can be associated with the different views of quality, i.e. “Users”,
“Managers” or “Developers”.
4 Abbreviated terms
BMT benchmark test
COTS commercial off-the-shelf
FOSS free/open-source software
GUI graphical user interface
5 Overview of evaluation and selection of software engineering tools
5.1 Introduction of the evaluation and selection of software engineering tools
This document defines both a set of processes and a structured set of software engineering tool
characteristics for use in the technical evaluation and the ultimate selection of a software engineering
tool. It follows the software product evaluation model defined in ISO/IEC 25041.
This document adopts the general model of software product quality characteristics and sub-
characteristics defined in ISO/IEC 25010 and extends these when the software product is a software
engineering tool; it provides product characteristics unique to software engineering tools as described
in 10.2 to 10.4. This larger set of characteristics is then organized into three groups; they are
characteristics related to software engineering tool usage functionality, general quality and not related
to quality.
NOTE The capabilities of software engineering tools are defined in a series of standards for each tool area.
The objective of the technical evaluation process is to provide quantitative results on which the final
selection can be based. Measurement assigns numbers (or other ratings) to attributes of entities; a major
activity of evaluation is to obtain these measurements for use in selection. The final selection results
should aim to achieve objectivity, repeatability and impartiality. These objectives and the confidence in
the outcomes depend in part on the resources allocated to the overall evaluation and selection process.
The user of this document is asked to deal with these issues at an early stage.
© ISO/IEC 2017 – All rights reserved 3

---------------------- Page: 9 ----------------------
ISO/IEC 20741:2017(E)

5.2 Framework of the evaluation and selection of software engineering tools
This subclause illustrates an overview of the evaluation and selection of software engineering tools
discussed in this document as shown in Figure 1. Evaluation and selection of software engineering
tools includes four major processes:
— preparation process;
— structuring process;
— evaluation process; and
— selection process.
In the process diagrams Figures 1 to 5, round corner rectangle is process/activity, normal rectangle
is outcome of the process/activity and dashed rectangle is a plan which is referred by each process
without reference. Also, a solid arrow shows data flow and a dashed arrow shows control flow of
transition between processes/activities.
Figure 1 — Framework of evaluation and selection of software engineering tools
A key process is the structuring of a set of requirements against which candidate software engineering
tools are to be evaluated and upon which selection decisions are based. The software engineering tool
characteristics defined in 10.2 to 10.4 form the basis for requirements structuring and play a central
role in the overall process.
5.3 General process considerations
There are several considerations that apply to the processes described in this document on a global
basis. The intent is for the user of this document to tailor its application in such a way as to maximize
the probability of a successful evaluation and selection process, and minimize its cost and risk.
5.3.1 Sequencing of processes
This document does not impose the sequence of process activities described above and in the following
clauses. It is up to the organization to select the relevant processes and activities needed to meet its
evaluation and selection goals.
The organization decides which to employ, in what sequence, and with what degree of parallelism. The
sequencing of the processes’ activities is then documented in an evaluation project plan.
4 © ISO/IEC 2017 – All rights reserved

---------------------- Page: 10 ----------------------
ISO/IEC 20741:2017(E)

5.3.2 Reducing cost and risk
In general, organizations which apply this document want to minimize the cost of the entire evaluation
and selection process to the extent possible, while maintaining the level of effort necessary to select
the most appropriate software engineering tool(s) for their use. These objectives can be addressed by
minimizing the number of tools evaluated, minimizing the cost of evaluating specific tools, and ensuring
that the formality of the process is appropriate to the organization.
The activities of software engineering tool information gathering and identifying final candidates for
selection (see Clause 9) effectively allow the user of this document to screen the available tools against
the organization’s needs, and eliminate from consideration tools which do not, or are not likely to,
substantially address the organization’s needs.
The organization can be unable to find any tool which appears likely to sufficiently meet its needs. In
such a case, the stated needs themselves should be re-examined, and if they are found to accurately
reflect the organization’s actual requirements for technology improvement, the overall evaluation
and selection process can be ab
...