ISO/IEC 14598-4:1999

INTERNATIONAL ISO/IEC
STANDARD 14598-4
First edition
1999-10-01
Software engineering — Product
evaluation —
Part 4:
Process for acquirers
Ingéniérie du logiciel — Évaluation du produit —
Partie 4: Procédé pour les acquéreurs
Reference number
ISO/IEC 14598-4:1999(E)
©
ISO/IEC 1999

---------------------- Page: 1 ----------------------
ISO/IEC 14598-4:1999(E)
PDF disclaimer
This PDF file may contain embedded typefaces. In accordance with Adobe's licensing policy, this file may be printed or viewed but shall not
be edited unless the typefaces which are embedded are licensed to and installed on the computer performing the editing. In downloading this
file, parties accept therein the responsibility of not infringing Adobe's licensing policy. The ISO Central Secretariat accepts no liability in this
area.
Adobe is a trademark of Adobe Systems Incorporated.
Details of the software products used to create this PDF file can be found in the General Info relative to the file; the PDF-creation parameters
were optimized for printing. Every care has been taken to ensure that the file is suitable for use by ISO member bodies. In the unlikely event
that a problem relating to it is found, please inform the Central Secretariat at the address given below.
© ISO/IEC 1999
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means, electronic
or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the address below or ISO's member body
in the country of the requester.
ISO copyright office
Case postale 56 � CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 734 10 79
E-mail copyright@iso.ch
Web www.iso.ch
Printed in Switzerland
ii © ISO/IEC 1999 – All rights reserved

---------------------- Page: 2 ----------------------
ISO/IEC 14598-4:1999(E)
Contents
1 Scope .1
2 Conformance.1
3 Normative references .2
4 Terms and definitions .2
5 Software product evaluation - General considerations.2
5.1 Correlation between evaluation and acquisition processes.2
5.2 Inputs to the evaluation process.3
5.2.1 System requirements .3
5.2.2 Integrity level requirements.4
5.2.3 Software requirements specification.4
5.2.4 Evaluations performed by others.5
5.3 Tailoring.5
6 Evaluation during acquisition of “off-the-shelf” software products.6
6.1 Step 1 - Establish evaluation requirements .7
6.1.1 Establish the purpose and scope of the evaluation.7
6.1.2 Specify evaluation requirements .8
6.2 Step 2 - Specify the evaluation.9
6.2.1 Select metrics.9
6.2.2 Select the evaluation methods.10
6.2.3 Evaluations performed by others.11
6.3 Step 3 - Design the evaluation.11
6.4 Step 4 - Execute the evaluation.13
6.4.1 Execute the evaluation methods.13
6.4.2 Analyze the evaluation results .13
6.4.3 Draw conclusions .14
7 Evaluation during acquisition of custom software and modifications to existing software .14
7.1 Step 1 - Establish evaluation requirements .15
© ISO/IEC 1999 – All rights reserved iii

---------------------- Page: 3 ----------------------
ISO/IEC 14598-4:1999(E)
7.2 Step 2 - Specify the evaluation.15
7.3 Step 3 - Design the evaluation.15
7.4 Step 4 - Execute the evaluation.15
Annex A (informative) Definitions from other standards .16
Annex B (informative) Tables.21
Annex C (informative) Figures .25
Annex D (informative) Evaluation methods.27
Annex E (informative) Example of staged evaluation process.32
Bibliography .34
iv © ISO/IEC 1999 – All rights reserved

---------------------- Page: 4 ----------------------
ISO/IEC 14598-4:1999(E)
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical Commission)
form the specialized system for worldwide standardization. National bodies that are members of ISO or IEC
participate in the development of International Standards through technical committees established by the
respective organization to deal with particular fields of technical activity. ISO and IEC technical committees
collaborate in fields of mutual interest. Other international organizations, governmental and non-governmental, in
liaison with ISO and IEC, also take part in the work.
International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 3.
In the field of information technology, ISO and IEC have established a joint technical committee, ISO/IEC JTC 1.
Draft International Standards adopted by the joint technical committee are circulated to national bodies for voting.
Publication as an International Standard requires approval by at least 75 % of the national bodies casting a vote.
Attention is drawn to the possibility that some of the elements of this part of ISO/IEC 14598 may be the subject of
patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights.
International Standard ISO/IEC 14598-4 was prepared by Joint Technical Committee ISO/IEC JTC 1, Information
technology, Subcommittee SC 7, Software engineering.
ISO/IEC 14598 consists of the following parts under the general title Software engineering — Product evaluation:
� Part 1: General overview
� Part 2: Planning and management
� Part 3: Process for developers
� Part 4: Process for acquirers
� Part 5: Process for evaluators
� Part 6: Documentation of evaluation modules
Annexes A to E of this part of ISO/IEC 14598 are for information only.
© ISO/IEC 1999 – All rights reserved v

---------------------- Page: 5 ----------------------
ISO/IEC 14598-4:1999(E)
Introduction
Software has become increasingly pervasive. The demand for added software functionality and faultless software
products has grown as more processes are automated to take advantage of the power of the computer. Today's
modern systems are so complex, that they are unable to perform their functions without software. The use of
commercially available “off-the-shelf” software products is accelerating as the variety of available products grows
and the rapid evolution of software engineering technology reduces reliance on custom-coded software. The
object-oriented development approach, which is based on the development of an application system through the
extension of existing libraries of self-contained units, has also reduced requirements for custom-coded software.
This has led to intense focus on concomitant software product quality or self-contained software unit quality.
Development of custom software is prone to rework as a result of failure to meet user requirements. The use of
custom software may also require a larger than anticipated effort with respect to deployment, implementation,
training, and maintenance support activities. Acquisition of commercial “off-the-shelf” software products, or, reuse
of in-house existing software products, is also not without risk. Problems can be encountered because the “off-the-
shelf” software products may require customizing; testing and analysis requirements may be large; product
maintenance and support is doubtful when the product becomes obsolete or revised; it may be difficult to integrate
software products into larger systems; and the quality of the product may not be consistent with the required quality
of the target system.
Commercial “off-the-shelf” software products are extremely varied. They can be:
� a) used as stand-alone products (i.e., payroll, accounting software, consumer software or 'shrink-wrapped
software' [i.e., word-processing software, spreadsheets]);
� b) integrated as components into a larger system which consists of other software and hardware
components (i.e., operating system, relational data base management system, graphical users interface
[GUI]);
� c) embedded in hardware (i.e., communication data link, programmable array logic [PAL]);
� d) embedded as part of a configurable software/hardware system that can be used for the development of a
specific application (i.e., distributed control system);
� e) CASE tools used to support the software development and maintenance process (i.e., compilers,
configuration management tools).
Errors in stand-alone software products can impact productivity, cause financial loss, or cause unnecessary rework.
Software components can be difficult to integrate, affect the reliability of the overall system, or be incompatible with
system objectives. CASE tools may introduce an error into a product under development or be difficult to use.
It is therefore essential to be able to evaluate the quality of software products during acquisition, or when making a
decision on reusing an existing software product or component. Evaluation may be used to accept or reject a single
product, or to select one product, from among alternative products, that meets the quality requirements established
for the target application. The level of rigor of the evaluation process is necessarily commensurate with the integrity
requirements for the product. The highest level of rigor is required when performing evaluation of software products
that are mission critical.
vi © ISO/IEC 1999 – All rights reserved

---------------------- Page: 6 ----------------------
INTERNATIONAL STANDARD © ISO/IEC ISO/IEC 14598-4 :1999(E)
Software engineering — Product evaluation — Part 4: Process for
acquirers
1 Scope
This part of ISO/IEC 14598 contains requirements, recommendations and guidelines for the systematic
measurement, assessment and evaluation of software product quality during acquisition of “off-the-shelf” software
products, custom software products, or modifications to existing software products. It uses the software quality
model described in ISO/IEC 9126-1; expands on the general process for evaluating software quality that is defined
in ISO/IEC 14598-1; and uses the process for acquisition that is defined in ISO/IEC 12207. It can be used in
conjunction with ISO/IEC 12119, ISO/IEC 14598-2 (new), ISO/IEC 14598-3 (new) and ISO/IEC 14598-6. The steps
of the evaluation process are similar between this part of ISO/IEC 14598 and ISO/IEC 14598-5, but the context of
use is quite different. In the case that acquirers entrust second or third parties with evaluations, ISO/IEC 14598-5 is
required to be applied. In the case that acquirers require third party testing of software packages against the quality
requirements for the package, ISO/IEC 12119 may be applied.
The evaluation process described in this part of ISO/IEC 14598 also helps to meet the objectives of deciding on the
acceptance of a single product, or for selecting a product from among alternate products. The evaluation process
may be tailored to the nature and integrity level of the application. It is also sufficiently flexible to accommodate the
wide range of forms and uses of software products in a cost-effective manner.
This part of ISO/IEC 14598 is intended for, but not limited to, project managers, system engineers, development
and maintenance software engineering staff and end users who plan to acquire software products, and also
suppliers who provide such products.
The target software products of the evaluation process in this part of ISO/IEC 14598 can be integrated into a larger
system as components or can be used stand-alone. They are classified as:
a) Commercial off-the-shelf software products;
b) Existing software products developed or acquired for other applications, or for a wide range of common
applications;
c) Custom software products or modifications to existing software products.
The evaluation process that is defined in this part is also applicable to CASE tools. Because evaluation of CASE
tools is specifically addressed in ISO/IEC 14102, CASE tools are considered out of scope of this part of ISO/IEC
14598.
ISO/IEC 14598-4 is designed to work in partnership with other standards. For systems with high integrity
requirements, additional requirements may be included in the evaluation process described in ISO/IEC 14598-4,
that are derived from sector-specific standards, e.g., IEC 880, DOA-167A , MOD-55, etc.
2 Conformance
Because of the freedom of choice afforded to the user by the general nature of its recommendations, a simple
claim of compliance with this part of ISO/IEC 14598 is not valid. Any organization imposing this part of ISO/IEC
14598 as a condition of trade is responsible for specifying and making public the evaluation process that meets the
mandatory objectives specified in 6.1.1. The specified evaluation process constitutes the terms for compliance for a
given application of this part of ISO/IEC 14598. All activities of clauses 6 and 7 shall be considered for applicability.
1

---------------------- Page: 7 ----------------------
© ISO/IEC
ISO/IEC 14598-4:1999(E)
Requirements on the evaluation process can also be established contractually during execution of the acquisition
process. Compliance with the evaluation process described in this part of ISO/IEC 14598 is then easily established.
3 Normative references
The following normative documents contain provisions which, through reference in this text, constitute provisions of
this part of ISO/IEC 14598. For dated references, subsequent amendments to, or revisions of, any of these
publications do not apply. However, parties to agreements based on this part of ISO/IEC 14598 are encouraged to
investigate the possibility of applying the most recent editions of the normative documents indicated below. For
undated references, the latest edition of the normative document referred to applies. Members of ISO and IEC
maintain registers of currently valid International Standards.
ISO/IEC 9126-1, Information technology - Software quality characteristics and metrics – Part 1: Quality
1)
characteristics and subcharacteristics.
ISO/IEC 12207:1995, Information technology - Software life cycle processes.
ISO/IEC 14598-1:1999, Information technology – Software product evaluation - Part 1: General overview.
ISO/IEC 14598-5:1998, Information technology – Software product evaluation - Part 5: Process for evaluators.
ISO/IEC 15026:1998, Information technology – System and software integrity levels.
4 Terms and definitions
For the purposes of this part of ISO/IEC 14598, the following definitions apply. Key definitions from other standards
used in this part of ISO/IEC 14598 are reproduced in Annex A for convenient reference.
4.1
commercial-off-the-shelf software (COTS)
software defined by a market-driven need, commercially available, and whose fitness for use has been
demonstrated by a broad spectrum of commercial users
NOTE See also definition in IEEE Std 1062-1993.
4.2
custom software
software developed for a specific application from a user requirements specification
4.3
existing software
software that is already developed and available; is usable either “as is” or with modifications; and which is
provided by the supplier, acquirer, or a third party
NOTE See also definition of “off-the-shelf product” [ISO/IEC 12207: 1995].
5 Software product evaluation - General considerations
5.1 Correlation between evaluation and acquisition processes
The acquisition process activities (defined in ISO/IEC 12207) summarized below, are combined with the general
evaluation process activities (defined in ISO/IEC 14598-1) in Clauses 6 and 7. Clause 6 focuses on the application
of the evaluation of end product quality during acquisition of COTS products. Clause 7 focuses on the application of
the evaluation process during acquisition of custom software or modifications to existing software.
1)
To be published. Until this part is published ISO/IEC 9126 :1991 should be used.
2

---------------------- Page: 8 ----------------------
© ISO/IEC
ISO/IEC 14598-4:1999(E)
a) Initiation - identification of software requirements for the product to be acquired, the acquisition plan, and the
acceptance strategy and criteria;
b) Request-for-proposal (-tender) preparation - specification and documentation of acquisition requirements;
c) Contract preparation and update - supplier selection, contract preparation and negotiation, and contract
change control;
d) Supplier monitoring - evaluation activities performed during contract execution leading to acceptance and
delivery of the software product;
e) Acceptance and completion - activities performed during product acceptance and delivery of the final software
product.
Note, that, the general evaluation process defined in ISO/IEC 14598-1, is not defined as a process in ISO/IEC
12207, but as an elementary function equivalent to the “check” portion of the plan-do-check-act (PDCA) cycle that
is implemented by each life-cycle process. However, the general evaluation process may be implemented in any
ISO/IEC 12207 process (i.e., development, maintenance, acquisition, validation); it is therefore at a different level of
abstraction from the sense of “process” used in ISO/IEC 12207.
This distinction is important when implementing the general evaluation process. The acquirer needs to define both
the evaluation process and the acquisition process that he/she will follow to achieve the evaluation requirements
during acquisition. In the context of larger system development, the acquisition and evaluation activities to be
followed need to be integrated with other development and integration activities, and identified in a project
measurement plan as specified in ISO/IEC 14598-2 (in preparation); i.e., specific acquisition implementation
considerations for evaluation include the following considerations:
� a software requirements specification required for evaluation can form the basis for acquisition requirements
required for the request-for-proposal (-tender);
� separate preliminary evaluation activities may be needed to preselect software products and suppliers;
� supplier and product information requirements for evaluation need to be specified in the acquisition
requirements or during contract preparation;
� evaluation activities can be executed as part of proposal evaluation, during monitoring of contract execution,
as part of product development, as part of formal product acceptance, or after product delivery.
5.2 Inputs to the evaluation process
5.2.1 System requirements
The starting point for determining evaluation requirements for target software begins with the overall system
requirements. The system requirements identify the user, user goals, tasks and characteristics, including the
environment in which the product is to be used, in addition to functional and other requirements for the product or
system. They form the basis for subsequent system architecture design, specification of software requirements,
and software architecture design. Relevant legal and regulatory requirements need to be identified at this stage as
they impact on the rigor and formality of the acquisition and evaluation processes.
During system requirements decomposition and design, system requirements are allocated to hardware and
software configuration items, and to user operations including system procedures. Design activities during a system
development life-cycle result in subsequent decisions to acquire or reuse “off-the-shelf” software products. Some of
the evaluation work is actually a part of these design activities, since it plays a role in the decision making process.
Evaluation of software products to be acquired is performed separately. During system integration and testing of
the end product, software configuration items are integrated with other software, and with the hardware
configuration items (Refer to ISO/IEC 12207). Figure 1 shows the larger systems engineering context for evaluation
and acquisition.
Candidates for use and acquisition in this part of ISO/IEC 14598 are software products which can be integrated into
a larger system as components or which can be used stand-alone. They are classified as:
3

---------------------- Page: 9 ----------------------
© ISO/IEC
ISO/IEC 14598-4:1999(E)
a) Commercial-off-the-shelf software products;
b) Existing software products developed or acquired for other applications, or for a wide range of common
applications;
c) Custom software products or modifications to existing software products.
In the case of software configuration items that are to be integrated into a larger system, software requirements
need to be defined for each item. In other cases, the system and the software configuration items coincide, and
may be considered equivalent.
Hardware configuration items to be acquired may contain software such as an operating system resident in
firmware (i.e. ROM, PROM). When the existing software forms an integral part of the hardware in this fashion it
usually needs to be evaluated with the hardware configuration item.
Initial
System Requirements
Mandate
System
Capture & System Design
Procedures
•user objectives & environment
•functional and other
requirements
•integrity level Non-Computer
Subsystems
Engineering
Computer Subsystems
Engineering
Subsystem
Integration &
Software Hardware
System
Engineering Engineering
Validation
System
Ergonomics & Product
Operation
Human Factors Evaluation &
Engineering Acquisition
Figure 1 — Systems engineering context for software product evaluation and acquisition
5.2.2 Integrity level requirements
If the software is critical in containing non-trivial safety, security, financial, environmental and societal risk of a
system within acceptable limits, its requred integrity level must have been established and properly documented
prior to procurement and evaluation. Guidance to the integrity level determination process is given in ISO/IEC
15026. The resulting integrity level determines how the software is dealt with in the evaluation process.
5.2.3 Software requirements specification
Software requirements shall be defined using an appropriate well defined quality model. For this purpose the
quality model and definitions in ISO/IEC 9126-1 should be used, unless there is a particular reason to use another
4

---------------------- Page: 10 ----------------------
© ISO/IEC
ISO/IEC 14598-4:1999(E)
model. This model defines six broad categories of characteristics of software in use: functionality, reliability,
usability, efficiency, maintainability and portability. These can be further broken down into subcharacteristics which
have measurable or assessable attributes.
Requirements should be defined in terms of external metrics (external metrics are defined in ISO/IEC 9126-2)
which directly relate to user needs and should be documented in a requirements specification. Documenting the
user needs can vary from producing an informal list of required functional and performance requirements to
preparing a complete requirements specification for the product (or system if the product is embedded). The
requirements specification may then form the basis for acquisition requirements used during the tendering step in
the acquisition process and the basis against which subsequent product evaluation is performed.
5.2.4 Evaluations performed by others
The scope of the present evaluation process can be reduced through access to the results of evaluation activities
performed by second or third parties as long as the results are trustworthy. Such evaluation activities can comprise
preexisting certifications, product evaluations and/or process assessments. For example:
� software engineering processes for product development may be standardized to meet the requirements of
ISO/IEC 12207, ISO 9000-3, or other sector-specific standards;
� the supplier’s quality system under which the software is developed may be certified to ISO 9001 requirements
by a third party;
� the software product may be evaluated by second or third parties to ISO/IEC 14598-5 or ISO/IEC 12119
requirements;
� the supplier’s software process capability for acceptable product development may be assessed by third
parties to ISO/IEC 15504-8 (in preparation);
� the software may be functionally evaluated as part of a larger system development phase;
� the software product may have been previously evaluated for another application with different integrity
requirements;
� evaluations on the product may have been performed by other parties within the organization through informal
or formal evaluation activities.
The additional costs and time required to obtain and interpret external evaluation results for the target application
may affect the feasibility of this method. It may still be necessary to consult the with the evaluator or supplier in
order to attain adequate confidence in the results of others.
NOTE Evaluation results of the supplier software engineering process, supplier quality system, or supplier capability alone are
not sufficient criteria to demonstrate that a software product contains the required quality characteristics. Other product
evaluation methods [e.g., such as those that specifically measure factors and attributes of quality that are appropriate to the
requirements of the end-user] need to be executed.
5.3 Tailoring
The evaluation process can be applied to a wide range of acquisition requirements, integrity requirements, and
evaluator objectives. For example:
� acquirers of software packages may wish to evaluate a software package using only ISO/IEC 12119;
� acquirers of software products may use ISO/IEC 14598-5 for independent evaluation;
� a small or sole acquirer may require a less formal evaluation process with minimum documentation of the
evaluation;
� for consumer software the evaluation process objective may simply be to select, test and acquire one product
from among a number of similar products. The formal acquisition process is then reduced to outright purchase,
and does not include contract preparation.
5

---------------------- Page: 11 ----------------------
© ISO/IEC
ISO/IEC 14598-4:1999(E)
The evaluation process should have the flexibility to accommodate the uniqueness of each application, to avoid
unnecessary work or work that adds no value, and to provide a practical means of establishing the requisite
confidence in the sof
...