ISO/IEC 20246:2017

INTERNATIONAL ISO/IEC
STANDARD 20246
First edition
2017-02
Software and systems engineering —
Work product reviews
Ingénierie du logiciel et des systèmes — Revue des produits de travail
Reference number
ISO/IEC 20246:2017(E)
©
ISO/IEC 2017

---------------------- Page: 1 ----------------------
ISO/IEC 20246:2017(E)

COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2017, Published in Switzerland
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized otherwise in any form
or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior
written permission. Permission can be requested from either ISO at the address below or ISO’s member body in the country of
the requester.
ISO copyright office
Ch. de Blandonnet 8 • CP 401
CH-1214 Vernier, Geneva, Switzerland
Tel. +41 22 749 01 11
Fax +41 22 749 09 47
copyright@iso.org
www.iso.org
ii © ISO/IEC 2017 – All rights reserved

---------------------- Page: 2 ----------------------
ISO/IEC 20246:2017(E)

Contents Page
Foreword .v
Introduction .vi
1 Scope . 1
2 Normative references . 1
3 Terms and definitions . 1
4 Conformance . 3
4.1 Intended usage . 3
4.2 Full conformance. 3
4.3 Tailored conformance . 3
5 Work product reviews. 4
5.1 Overview . 4
5.2 Review attributes . 4
5.3 Review types . 4
6 Work product review process . 5
6.1 Overview . 5
6.2 Purpose . 5
6.3 Outcomes . 5
6.4 Activities and tasks . 6
6.4.1 Planning . 6
6.4.2 Initiate review . 6
6.4.3 Individual review . . 7
6.4.4 Issue communication and analysis . 7
6.4.5 Fixing and reporting . 8
6.5 Information items . 8
7 Review techniques . 8
7.1 Overview . 8
7.2 Individual reviewing techniques . 9
7.2.1 Overview . 9
7.2.2 Ad hoc reviewing . 9
7.2.3 Checklist-based reviewing . 9
7.2.4 Scenario-based reviewing . 9
7.2.5 Perspective-based reading (PBR).10
7.2.6 Role-based reviewing .11
7.3 Issue analysis techniques .11
7.3.1 Overview .11
7.3.2 Individual analysis .11
7.3.3 Review meeting techniques .11
7.3.4 Group decision making .12
Annex A (normative) Review documentation .13
Annex B (informative) Review documentation examples .21
Annex C (informative) Review attributes .26
Annex D (informative) Review types .30
Annex E (informative) Mapping to IEEE 1028-2008 .34
Annex F (informative) Review selection based on work product .35
Annex G (informative) Reviews — Life cycle mapping .37
Annex H (informative) Review measurement and improvement .39
Annex I (informative) Tool support .41
© ISO/IEC 2017 – All rights reserved iii

---------------------- Page: 3 ----------------------
ISO/IEC 20246:2017(E)

Bibliography .42
iv © ISO/IEC 2017 – All rights reserved

---------------------- Page: 4 ----------------------
ISO/IEC 20246:2017(E)

Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialized system for worldwide standardization. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organization to deal with particular fields of technical
activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international
organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the
work. In the field of information technology, ISO and IEC have established a joint technical committee,
ISO/IEC JTC 1.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for
the different types of document should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www .iso .org/ directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent
rights. Details of any patent rights identified during the development of the document will be in the
Introduction and/or on the ISO list of patent declarations received (see www .iso .org/ patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation on the meaning of ISO specific terms and expressions related to conformity assessment,
as well as information about ISO’s adherence to the World Trade Organization (WTO) principles in the
Technical Barriers to Trade (TBT) see the following URL: www . i so .org/ iso/ foreword .html.
This document was prepared by Technical Committee ISO/IEC JTC 1, Information technology,
Subcommittee SC 7, Software and systems engineering.
© ISO/IEC 2017 – All rights reserved v

---------------------- Page: 5 ----------------------
ISO/IEC 20246:2017(E)

Introduction
The purpose of this document is to provide an International Standard that defines work product
reviews, such as inspections, reviews and walkthroughs that can be used at any stage of the software
and systems life cycle. It can be used to review any system or software work product. This document
defines a generic process for work product reviews that can be configured based on the purpose of the
review and the constraints of the reviewing organization. The intent is to describe a generic process
that can be applied both efficiently and effectively by any organization to any work product.
The main objectives of reviews are to detect issues, to evaluate alternatives, to improve organizational
and personal processes, and to improve work products. When applied early in the life cycle, reviews are
typically shown to reduce the amount of unnecessary rework on a project. The work product review
techniques presented in this document can be used at various stages of the generic review process to
identify defects and evaluate the quality of the work product.
Review documents that are produced during work product reviews are defined in Annex A.
vi © ISO/IEC 2017 – All rights reserved

---------------------- Page: 6 ----------------------
INTERNATIONAL STANDARD ISO/IEC 20246:2017(E)
Software and systems engineering — Work product reviews
1 Scope
This document establishes a generic framework for work product reviews that can be referenced and
used by all organizations involved in the management, development, test and maintenance of systems
and software. It contains a generic process, activities, tasks, review techniques and documentation
templates that are applied during the review of a work product. A work product is any artefact
produced by a process. This document defines work product reviews that can be used during any phase
of the life cycle of any work product. This document is intended for, but not limited to, project managers,
development managers, quality managers, test managers, business analysts, developers, testers,
customers and all those involved in the development, testing and maintenance of systems and software.
2 Normative references
The following documents are referred to in the text in such a way that some or all of their content
constitutes requirements of this document. For dated references, only the edition cited applies. For
undated references, the latest edition of the referenced document (including any amendments) applies.
ISO/IEC/IEEE 24765, Systems and software engineering — Vocabulary
3 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO/IEC/IEEE 24765 and the
following apply.
ISO and IEC maintain terminological databases for use in standardization at the following addresses:
— IEC Electropedia: available at http:// www .electropedia .org/
— ISO Online browsing platform: available at http:// www .iso .org/ obp
3.1
ad hoc reviewing
unstructured independent review technique
3.2
author check
informal review performed by the author of the work product
3.3
buddy check
informal review performed independently by a colleague of the author
3.4
checklist-based reviewing
review technique guided by a list of questions or required attributes
3.5
formal review
form of review that follows a defined process with formal documented output
3.6
informal review
form of review that does not follow a defined process and has no formal documented output
© ISO/IEC 2017 – All rights reserved 1

---------------------- Page: 7 ----------------------
ISO/IEC 20246:2017(E)

3.7
informal group review
informal review performed by three or more persons
3.8
inspection
formal review of a work product to identify issues, which uses defined team roles and measurement to
improve the review process
[7]
EXAMPLE Fagan Inspections are a specific type of inspection and code inspections are used to review
program source code.
3.9
issue
observation that deviates from expectations
EXAMPLE Potential defect, improvement or point needing clarification.
3.10
milestone review
formal review of a work product and supporting evidence used to determine its acceptability for use in
the next stage of development or for delivery
Note 1 to entry: The requirement for this form of review is normally specified in the project plan.
3.11
page-by-page reviewing
technique where reviewers review a work product in a sequential order
3.12
pair review
informal review of a work product performed by two suitably qualified people other than the author
working together
3.13
peer desk check
informal review where the author and a colleague walk through a work product
3.14
peer review
review of work products performed by others qualified to do the same work
3.15
perspective-based reading
form of role-based reviewing that uses checklists and involves the creation of prototype deliverables to
check the completeness and other quality characteristics of the work product
3.16
role-based reviewing
technique where reviewers review a work product from the perspective of different stakeholder roles
EXAMPLE Typical stakeholder roles include specific user types, such as work product maintainer, tester and
developer.
3.17
scenario-based reviewing
technique where the review is guided by determining the ability of the work product to address specific
scenarios
2 © ISO/IEC 2017 – All rights reserved

---------------------- Page: 8 ----------------------
ISO/IEC 20246:2017(E)

3.18
technical review
formal peer review of a work product by a team of technically-qualified personnel that examines the
suitability of the work product for its intended use and identifies discrepancies from specifications and
standards
Note 1 to entry: Technical review may also provide recommendations of alternatives and examination of various
alternatives.
3.19
walkthrough
formal review in which an author leads members of the review through a work product, and the
participants ask questions and make comments about possible issues
3.20
work product
artefact produced by a process
EXAMPLE Project plan, requirements specification, design documentation, source code, test plan, test
meeting minutes, schedules, budgets, and incident reports.
Note 1 to entry: A subset of the work products can be baselined to be used as the basis of further work and some
will form the set of project deliverables.
4 Conformance
4.1 Intended usage
The normative requirements in this document are contained in Clause 6 and Annex A. It is recognized
that particular projects or organizations may not need to use all of the techniques defined by this
document. Therefore, implementation of this document typically involves selecting a set of techniques
suitable for the project or organization. There are two ways that an organization or individual can claim
conformance to the provisions of this document. The organization or individual shall assert whether
full or tailored conformance to this document is claimed.
4.2 Full conformance
Full conformance is achieved by demonstrating that all of the requirements (i.e. “shall” statements)
of the work product review process defined in Clause 6 and the review documentation annex of this
document have been satisfied.
4.3 Tailored conformance
When this document is used as a basis for establishing a review process that does not qualify for full
conformance, the subset of activities for which tailored conformance is claimed, is recorded. Tailored
conformance is achieved by demonstrating that all of the requirements (i.e. “shall” statements) for the
recorded subset of activities have been satisfied.
Where tailoring occurs, justification shall be provided (either directly or by reference), whenever an
activity defined in Clause 6 of this document is not followed. All tailoring decisions shall be recorded
with their rationale, including the consideration of any applicable risks. Tailoring decisions shall be
agreed by the relevant stakeholders.
© ISO/IEC 2017 – All rights reserved 3

---------------------- Page: 9 ----------------------
ISO/IEC 20246:2017(E)

5 Work product reviews
5.1 Overview
Work product reviews are performed on many projects, typically as a means of contributing to the
early detection of defects, so that these defects can be removed as early as possible thus reducing
unnecessary rework. In practice, reviews are performed for a variety of purposes in addition to defect
detection (examples are listed in C.1.2.1).
Reviews can be classified in a number of ways. In this document, reviews are classified as either formal
or informal. Many review techniques can be used over the course of a review, such as role-based
reviewing for individual review and checklist-based reviewing during a review meeting.
The generic process for conducting work product reviews (defined in Clause 6) includes a number of
selectable attributes (including review techniques). This allows users to configure their specific review
type according to their unique situation. These attributes are described in detail in Annex C. This
configuration of the generic process allows users to define reviews that suit their purpose while still
conforming to their constraints in the most effective and efficient manner, rather than forcing them to
choose a specific named review type that they cannot practically use in full.
Historically in the literature a number of distinct review types have been defined but some differ
only in the extent to which a particular attribute is emphasized (these types are listed in 5.3 and the
mapping between the characteristics and review types is provided in Annex D). For example, some
believe the difference between inspection and technical review simply to be that inspection requires
process improvement.
5.2 Review attributes
The following is a list of review attributes that can be used to define the review to be performed.
Annex C provides more detail on each of the attributes.
— Purpose (see C.1.2.1);
— Roles (see C.1.2.2);
— Individual review techniques (see C.1.2.3);
— Optional activities (see C.1.2.4);
— Number of reviewers (see C.1.2.5);
— Planned number of reviews (see C.1.2.6);
— Formal reporting (see C.1.2.7);
— Training required (see C.1.2.8);
— Review improvement (see C.1.2.9);
— Entry and exit criteria (see C.1.2.10).
Annex F provides guidelines on the selection of review attributes for different work product types and
work product formats.
5.3 Review types
[13]
The following is a list of review types commonly referenced in the literature and found in IEEE 1028.
Annex E describes the alignment of the activities defined in this document with the procedures of
4 © ISO/IEC 2017 – All rights reserved

---------------------- Page: 10 ----------------------
ISO/IEC 20246:2017(E)

IEEE 1028-2008. Annex D provides more detail on each of the types and maps the relevant attributes
from 5.2 to the different review types.
— Author check;
— Buddy check;
— Informal group review;
— Inspection;
— Milestone review;
— Pair review;
— Peer desk check;
— Technical review;
— Walkthrough.
Annex G provides examples of how each review type can be used within specific software/systems
development life cycle models. Users of this document are not restricted to using the above review
types. They can also use hybrid types based on selected attributes applied to the generic review process
according to their needs.
6 Work product review process
6.1 Overview
The Work Product Review Process comprises activities for the review of work products (see Figure 1).
The process shown in Figure 1 is not always performed on “complete” work products, but can be
performed on parts of work products, and in this situation these activities will typically be invoked
a number of times to complete the review for a complete work product. Thus, the process shown in
Figure 1 can be applied more than once on a single work product.
Figure 1 — Work Product Review Process
6.2 Purpose
The purpose of the Work Product Review Process is to provide a structured but flexible framework
from which review processes (both formal and informal) may be tailored for specific contexts and
purposes.
6.3 Outcomes
As a result of the successful implementation of the Work Product Review Process:
a) defects/issues in the work product are identified;
b) quality characteristics of the work product are evaluated;
NOTE A list of quality characteristics can be found in the ISO/IEC 25000 series of standards.
© ISO/IEC 2017 – All rights reserved 5

---------------------- Page: 11 ----------------------
ISO/IEC 20246:2017(E)

c) reviewers have gained knowledge about the work product;
d) consensus on decisions made has been reached;
e) new ideas have been generated;
f) updates to the work product are made;
g) participants have identified potential improvements in their working practices.
6.4 Activities and tasks
The person(s) responsible for the work product review shall implement the following activities and
tasks in accordance with applicable organization policies and procedures with respect to the Work
Product Review Process.
6.4.1 Planning
This activity consists of the following tasks:
a) The scope of the review, which comprises the purpose, the work product to be reviewed, quality
characteristics to be evaluated, areas to focus on, exit criteria, supporting information such as
standards, effort and the timeframes for the review, shall be defined.
NOTE 1 The work product to be reviewed can be part of a larger work product.
EXAMPLE 1 Areas to focus on can be specific features, non-functional attributes or selected pages.
b) The review characteristics shall be identified and agreed.
EXAMPLE 2 Review characteristics can include review activities, roles, effort, individual review
techniques and checklists.
NOTE 2 The responsibility for identifying and agreeing the review characteristics usually involves roles
such as the review leader, management and reviews coordinator as defined in C.1.2.2.
c) The review participants, along with their expected roles, shall be identified and agreed.
6.4.2 Initiate review
This activity consists of the following tasks:
a) Required review materials shall be distributed to review participants.
EXAMPLE Review materials can include, but are not limited to, the work product, checklists, review
guidelines and the baseline specification.
b) The review leader shall communicate the scope and characteristics of the review to the review
participants.
c) The review leader shall communicate the roles, responsibilities and focus to each review
participant.
d) The author (or a suitably qualified person) may describe the work product under review.
NOTE 1 Tasks b), c), and d) can be performed at an overview meeting.
NOTE 2 The decision to hold an overview meeting typically depends on factors such as whether the
reviewers:
— have previously participated in or been trained in formal reviews;
— know and understand the review process to be used;
6 © ISO/IEC 2017 – All rights reserved

---------------------- Page: 12 ----------------------
ISO/IEC 20246:2017(E)

— understand its objectives (e.g. documenting issues versus proposing resolutions);
— are familiar with the concept of assigned roles, the requirements of their specific roles, the classification
system for issues and the forms and tools (see Annex I) to be used in the process;
— require additional background information about the work product or its context.
e) Training for reviewers may be arranged.
6.4.3 Individual review
This activity consists of the following tasks:
a) Each reviewer shall perform a review to identify issues with the work product.
NOTE 1 Issues will typically be documented in an Issue Log (as described in A.2) and will be classified in
terms of severity.
NOTE 2 Issues can be supported by proposed changes.
6.4.4 Issue communication and analysis
This activity consists of the following tasks:
a) Identified issues shall be communicated.
NOTE 1 If a review meeting is held, then issues can be voiced at the meeting or can be sent for collation
and prioritization prior to the meeting.
NOTE 2 If a review meeting is not held, then issues are typically sent to the individual performing the
analysis.
b) Previously identified issues, and any new issues identified during this activity, shall be analysed to
assign them a status based on the subsequent action to be taken on them.
EXAMPLE 1 Typical examples of issue status are “rejected”, “issue to be noted but no action” and
“issue to be addressed”.
c) Issues shall be assigned to an appropriate individual or team based on their status.
NOTE 3 In an informal review the assignment and status of an issue do not need to be documented.
EXAMPLE 2 This can include the assignment of issues to work product authors or individuals (or
teams) external to the review (where an issue relates to supporting documentation, such as an organization-
wide standard).
d) The quality characteristics of the work product under review shall be evaluated and, along with
other relevant criteria, used to make the revi
...