ISO/IEC 14598-2:2000

INTERNATIONAL ISO/IEC
STANDARD 14598-2
First edition
2000-02-01
Software engineering — Product
evaluation —
Part 2:
Planning and management
Ingénierie du logiciel — Évaluation du produit —
Partie 2: Planification et gestion
Reference number
ISO/IEC 14598-2:2000(E)
©
  ISO/IEC 2000

---------------------- Page: 1 ----------------------
ISO/IEC 14598-2:2000(E)
PDF disclaimer
This PDF file may contain embedded typefaces. In accordance with Adobe's licensing policy, this file may be printed or viewed but shall not
be edited unless the typefaces which are embedded are licensed to and installed on the computer performing the editing. In downloading
this file, parties accept therein the responsibility of not infringing Adobe's licensing policy. The ISO Central Secretariat accepts no liability in
this area.
Adobe is a trademark of Adobe Systems Incorporated.
Details of the software products used to create this PDF file can be found in the General Info relative to the file; the PDF-creation
parameters were optimized for printing. Every care has been taken to ensure that the file is suitable for use by ISO member bodies. In the
unlikely event that a problem relating to it is found, please inform the Central Secretariat at the address given below.
©  ISO/IEC 2000
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means, electronic
or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the address below or ISO's member body
in the country of the requester.
ISO copyright office
Case postale 56 • CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 734 10 79
E-mail copyright@iso.ch
Web www.iso.ch
Printed in Switzerland
ii © ISO/IEC 2000 – All rights reserved

---------------------- Page: 2 ----------------------
© ISO/IEC ISO/IEC 14598-2:2000(E)
Contents
Foreword iii
Introduction iv
1Scope 1
2Conformance 1
3 Normative references 1
4 Terms and definitions 2
5 Evaluation management concepts 2
6 Requirements and recommendations for supporting software evaluation 3
6.1 General 3
6.2 Management at organisational level 4
  6.2.1 Planning the use and improvement of the evaluation technology 4
6.2.2 Implementation of the evaluation technology 4
  6.2.3 Transfer of technology used for evaluation 5
6.2.4 Assessment of the technology used for the evaluation 5
6.2.5 Management of experiences 6
6.3 Support for Project Management 6
6.3.1 Support for Evaluation Planning 7
6.3.2 Ongoing promotion of the Quantitative Evaluation Plan 8
6.3.3 Supporting the evaluation projects 8
6.3.4 Collection of the evaluation results 8
Annex A (normative) Quantitative Evaluation Plan Template 9
Bibliography 12
iii

---------------------- Page: 3 ----------------------
ISO/IEC 14598-2:2000(E) © ISO/IEC
Foreword
ISO (the International Organization for Standardization) and IEC (The International
Electrotechnical Commission) form the specialized system for worldwide
standardization. National bodies that are members of ISO or IEC participate in the
development of International Standards through technical committees established by
the respective organization to deal with particular fields of technical activity. ISO and
IEC technical committees collaborate in fields of mutual interest. Other international
organizations, governmental and non-governmental, in liaison with ISO and IEC,
also take part in the work.
In the field of information technology, ISO and IEC have established a joint technical
committee, ISO/IEC JTC 1. Draft International Standards adopted by the joint
technical committee are circulated to national bodies for voting. Publication as an
International Standard requires approval by at least 75 % of the national bodies
casting a vote.
International Standard ISO/IEC 14598-2 was prepared by Joint Technical Committee
ISO/IEC JTC 1, Information technology, Subcommittee SC 7, Software engineering.
ISO/IEC 14598 consists of the following parts, under the general title Software
engineering — Product evaluation:
— Part 1: General overview
— Part 2: Planning and management
— Part 3: Process for developers
— Part 4: Process for acquirers
— Part 5: Process for evaluators
— Part 6: Documentation of evaluation modules
Annex A forms a normative part of this part of ISO/IEC 14598.
iv

---------------------- Page: 4 ----------------------
© ISO/IEC ISO/IEC 14598-2:2000(E)
Introduction
This part of ISO/IEC 14598 provides details about the planning and management
requirements which are associated with software product evaluation.
While this part of ISO/IEC 14598 is mainly concerned with product evaluation,
wherever it is relevant the corresponding process evaluation activities are also
discussed.
This part of ISO/IEC 14598 aims to clarify the requirements which should be
provided by the organisation in order to ensure the success of the evaluation. This
supporting function can be part of the organisation (e.g. a Technical Group), or a
specially created management function.
v

---------------------- Page: 5 ----------------------
©
INTERNATIONAL STANDARD  ISO/IEC ISO/IEC 14598-2:2000(E)
Software engineering — Product evaluation —
Part 2:
Planning and management
1  Scope
This part of ISO/IEC 14598 provides requirements, recommendations and guidelines for a supporting function which is responsible
for the management of software product evaluation and for technologies necessary for software product evaluation.
The role of the supporting function includes motivating people and training them for the evaluation activities, preparing appropriate
evaluation documents and methods, and responding to queries on evaluation technologies.
Main targets for evaluation support are software development, system integration and maintenance, including software acquisition,
at both project and organisation levels.
Technology management is related to the planning and management of a software evaluation process, metrics and tools. This
includes the management of development, acquisition, standardisation, control, transfer and feedback of evaluation technology
experiences within the organisation.
The intended users of this part of ISO/IEC 14598 are people who are responsible for
• managing the use of the evaluation technology,
• supporting software product evaluation,
• managing software development organisations,
or people in a quality assurance function. However, it is also applicable to managers involved in other software related activities.




2  Conformance

In order to conform to this part of ISO/IEC 14598, an organisation shall review all requirements and recommendations in clause 6,
to identify those which are applicable, and state which requirements and recommendations have not been implemented.




3  Normative references

The following normative documents contain provisions which, through reference in this text, constitute provisions of this part of
ISO/IEC 14598. For dated references, subsequent amendments to, or revisions of, any of these publications do not apply. However,
parties to agreements based on this part of ISO/IEC 14598 are encouraged to investigate the possibility of applying the most recent
editions of the normative documents indicated below. For undated references, the latest edition of the normative documents
referred to applies. Members of ISO and IEC maintain registers of currently valid International Standards.

ISO 8402:1994, Quality management and quality assurance — Vocabulary.

ISO/IEC 9126:1991, Information technology — Software product evaluation — Quality characteristics and guidelines for their
use.

ISO/IEC 9126-1, Information technology — Software product quality — Part 1: Quality model.



1

---------------------- Page: 6 ----------------------
©
ISO/IEC 14598-2:2000(E) ISO/IEC
ISO/IEC 9126-2, Information technology — Software product quality — Part 2: External metrics.

ISO/IEC 9126-3, Information technology — Software product quality — Part 3: Internal metrics.

ISO/IEC 14598-1:1999, Information technology — Software product evaluation — Part 1: General overview.

ISO/IEC 14598-5:1998, Information technology — Software product evaluation — Part 5: Process for evaluators.

ISO/IEC 14598-6, Software engineering — Product evaluation — Part 6: Documentation of evaluation modules.




4  Terms and definitions

For the purposes of this part of ISO/IEC 14598, the terms and definitions given in ISO 8402 and ISO/IEC 14598-1 and the
following apply.

4.1
evaluation technology (technology used for evaluation)
techniques, tools, metrics, measures and other technical information, used for evaluation

4.2

supporting function
an organisation responsible for assisting the software evaluation activities through the provision of technology, tools, experiences,
and management skills

4.3
techniques
methods and skills required to carry out a specific task



5  Evaluation management concepts

This part of ISO/IEC 14598 is applicable to the supporting function, which provides the organisation-wide support to all projects in
software development, software acquisition and third party evaluation organisations (see tables 1 and 2).




Table 1 — Software evaluation activities
DEVELOPED SOFTWARE ACQUIRED SOFTWARE
Development Evaluation Acquisition Evaluation
Activities Activities Activities Activities
The "deliverables" Evaluation of Dependent upon the Review specific output
are dependent upon specific "deliverables" selected acquisition of the acquisition
the chosen life cycle (output of the project) process process
e.g. System Requirements e.g. Review System e.g. Request for proposal; Review proposal request
Specification, Design Suppliers process Audit supplier's process
Systems Design Specification





2

---------------------- Page: 7 ----------------------
©
ISO/IEC ISO/IEC 14598-2:2000(E)
The main role of the supporting function should include

• acquisition of relevant international and national standards, technical information and, if required, expert support,
• development of suitable in-house standards and tools based upon the project’s and organisation's requirements,
• development of criteria for setting benchmarks for the evaluation,
• reviewing the effectiveness and quality of any software acquisition and development,
• collection and analysis of evaluation results and the dissemination of these within the organisation by the use of a
database,
• facilitation of technology transfer, based upon the experiences within the organisation and its respective evaluation
projects,
• support of evaluation projects and their respective project managers.

The supporting function can be external or internal with respect to the organisation which is evaluating the software.

If the supporting function is an internal part of the evaluating organisation, it can be within or outside the department which is
involved in the software evaluation. The specific roles of the supporting function and the evaluation projects are shown in table 2.
This table also shows the relationship between the activities of the supporting function and the evaluation projects.


Table 2 — Relationship between supporting function and evaluation projects
SUPPORTING FUNCTION EVALUATION PROJECTS
PROVIDES DEVELOP
* NEW TECHNOLOGY * PROJECT EXPERIENCE
* INTERNATIONAL/NATIONAL STANDARDS * EVALUATION EXPERIENCE
* EXPERTISE (CONSULTANCY) * PROJECT DATA
* TRAINING * EXPERIENCE WITH TECHNOLOGY
* ORGANISATION DATA BASE * FEED-BACK TO THE
* SUPPORT TO EVALUATION PROJECTS SUPPORTING FUNCTION
6  Requirements and recommendations for supporting software evaluation

6.1  General

The organisation shall develop a policy and plans for all evaluation activities. The responsibility of the supporting function shall
also be defined for all evaluation activities.

a) The following steps shall be followed when planning and executing software evaluation.

1) Define the objectives of the software evaluation.

2) Ensure that a Quantitative Evaluation Plan for all evaluation projects is developed. This plan may be subdivided
into lower level plans, subject to the complexity of the respective evaluation (see Annex A).

3) Enter project and/or product evaluation experiences into the organisation's database, to improve the
organisation's approach to software evaluation.

b) Organisations should carry out all their software evaluations in accordance with the following:

1) evaluate whether the software conforms to international, national or internal standards (if that is applicable),

2) ensure that the evaluation results can be quantified, clearly presented and traceable,

3) ensure that suitable and effective technology and best practices are used,
3

---------------------- Page: 8 ----------------------
©
ISO/IEC 14598-2:2000(E) ISO/IEC

4) ensure that the evaluation is carried out effectively,

5) ensure that plans and recommendations supporting all future evaluation activities are available.



6.2  Management at organisational level

Organisations that develop, acquire or evaluate software repeatedly shall have the overall evaluation responsibilities and quality
assurance activities clearly defined and incorporated into a plan.

NOTE - When implemented, this plan will help to improve the quality of the evaluation and ensure the best use of the available and relevant
technology.

Some organisations may choose to entrust the evaluation activities to a third party. This third party shall also manage the evaluation
technology in accordance with the following requirements and recommendations.



6.2.1  Planning the use and improvement of the evaluation technology

An overall plan for improving the software evaluation and its supporting techniques shall be made and implemented.

The plan should include the following:

a) Preparation of a policy statement

There shall be a policy stating the organisation's approach to the introduction, maintenance and improvement of software
quality evaluation.


b) Definition of the organisation’s objectives

The organisation’s objectives, which are to be achieved by the introduction, maintenance and improvement of software quality
evaluation technology, shall be defined.

c) Identification of the technology to be used

The software evaluation methods and techniques used in the organisation shall be assessed and identified in the plan. Any
deviation from the stated objectives shall be corrected.

d) Assignment of responsibilities for the management of the evaluation process

Clearly stated responsibility shall be assigned for the introduction, maintenance and ongoing improvement of the evaluation
process.

e) Identification of further improvements

The process and activities for investigating the availability and applicability of new technology shall be identified. This includes
conducting trials and evaluations, and introducing and maintaining new techniques.


6.2.2  Implementation of the evaluation technology

T he organisation shall
a) assess its own and externally available quality evaluation technology and shall determine its technology needs
and, if necessary, how any new technology can be acquired,
b) clarify and define the detailed requirements for acquiring or developing the evaluation technology according to
the results of the work described in a) above. These plans shall then be implemented,
c) define the process for adopting and operating the acquired evaluation technology.
4

---------------------- Page: 9 ----------------------
©
ISO/IEC ISO/IEC 14598-2:2000(E)
Any validated evaluation module should be maintained under configuration control, and documented as an Evaluation Module (see
ISO/IEC 14598-6). Otherwise it should be put into trial use for assessment.

The software evaluation process for an organisation shall be determined. If this is not available in-house, it shall be acquired.

In the case of acquisition

a) first, if international or national standards are available, the organisation should introduce these,

b) second, if well-known evaluation technology within academia or industry is available, the organisation should
consider introducing these,

c) finally, the organisation should consider developing the appropriate technology or contracting an external expert
a
...