ETSI EG 203 341 V1.1.1 (2016-10)

ETSI GUIDE
Core Network and Interoperability Testing (INT);
Approaches for Testing Adaptive Networks

2 ETSI EG 203 341 V1.1.1 (2016-10)

Reference
DEG/INT-00127
Keywords
conformance, interoperability, methodology
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: +33 4 92 94 42 00  Fax: +33 4 93 65 47 16

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
The present document can be downloaded from:
http://www.etsi.org/standards-search
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the only prevailing document is the
print of the Portable Document Format (PDF) version kept on a specific network drive within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
https://portal.etsi.org/TB/ETSIDeliverableStatus.aspx
If you find errors in the present document, please send your comment to one of the following services:
https://portal.etsi.org/People/CommiteeSupportStaff.aspx
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© European Telecommunications Standards Institute 2016.
All rights reserved.
TM TM TM
DECT , PLUGTESTS , UMTS and the ETSI logo are Trade Marks of ETSI registered for the benefit of its Members.
TM
3GPP and LTE™ are Trade Marks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.
ETSI
3 ETSI EG 203 341 V1.1.1 (2016-10)
Contents
Intellectual Property Rights . 5
Foreword . 5
Modal verbs terminology . 5
Introduction . 5
1 Scope . 7
2 References . 7
2.1 Normative references . 7
2.2 Informative references . 7
3 Definitions and abbreviations . 8
3.1 Definitions . 8
3.2 Abbreviations . 8
4 Definition of Adaptive Networks . 9
4.1 Basic Concept . 9
4.2 General Terminology . 11
4.2.1 Introduction. 11
4.2.2 Network States . 11
4.2.3 Static and stationary states . 12
4.2.4 State Transitions and Attractors . 12
4.3 Adaptive Networks as Network Under Test . 14
5 Entities and interactions . 15
5.1 Overview . 15
5.2 Effectors/Activitie s . 17
5.2.1 User-equivalent activities (type A1) . 17
5.2.1.1 Introduction . 17
5.2.1.2 Systems delivering the required functionality . 17
5.2.2 Structural or other activities (type A2) . 17
5.2.2.1 Introduction . 17
5.2.2.2 Systems delivering the required functionality . 17
5.2.3 Additional controls . 18
5.3 Information/Sensors . 18
5.3.1 Network performance from end user perspective (type I1) . 18
5.3.1.1 Introduction . 18
5.3.1.2 Systems delivering the required functionality . 18
5.3.2 Additional information about the network (type I2) . 19
5.3.3 Additional aspects of sensors . 19
6 Functional Targets . 19
6.1 Introduction . 19
6.2 Network stages . 19
6.3 Classes of functional targets . 21
6.4 Applicability of functional targets to network stages . 22
7 Generic Framework and Methods for Testing Adaptive Networks . 22
7.1 Basic Assumptions . 22
7.2 General aspects and related terminology . 23
7.3 Testing Process . 23
7.3.1 Introduction. 23
7.3.2 A1 based testing scenarios . 23
7.3.3 A2 based testing scenarios . 24
7.4 Evaluation of results . 25
Annex A: Relation to other work done in this field. 26
A.1 Introduction . 26
ETSI
4 ETSI EG 203 341 V1.1.1 (2016-10)
A.2 ISG NFV . 26
A.2.1 Group description . 26
A.2.2 Network Functions Virtualisation (NFV) . 26
A.3 NTECH AFI . 26
A.3.1 Group description . 26
A.3.2 GANA model overview . 27
A.3.3 Concepts of the Generic Test Framework for Testing Adaptive Functions . 27
History . 31

ETSI
5 ETSI EG 203 341 V1.1.1 (2016-10)
Intellectual Property Rights
IPRs essential or potentially essential to the present document may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (https://ipr.etsi.org/).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Foreword
This ETSI Guide (EG) has been produced by ETSI Technical Committee Core Network and Interoperability
Testing (INT).
Modal verbs terminology
In the present document "should", "should not", "may", "need not", "will", "will not", "can" and "cannot" are to be
interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
Introduction
The characteristics of "adaptive networks" such as virtualization, self-organization, self-configuration, self-
optimization, self-healing and self-learning, dynamic network slicing promise to offer huge advantages in future
networks. While technologies such as Network Functions Virtualisation (NFV), Self-Organizing Networks (SON),
Mobile Edge Computing (MEC) and Autonomic Management and Control (AMC) of Networks and Services may not
each exhibit all the characteristics they do have one thing in common: they are all dynamic rather than static, reacting to
dynamic traffic conditions, applications, service demands as well as to changes in the eco-system environment.
By incorporating one or several of the technologies mentioned above, Adaptive Networks (AN) have the ability to
automatically and dynamically manage and control network resources, configuration parameters or the network
structure, with limited human intervention, in order to meet functional targets or operational policies. However, to
achieve this type of autonomic behaviour, it has to be ensured that any modification that is performed automatically in
the network does not produce undesired effects, e.g. instability or lower performance with respect to the end-user
perspective.
Comprehensive testing, both on a general level as in type approvals and related to acceptance testing of a particular
deployment, is therefore even more important than it is for conventional networks. Due to the fact that the components
of an AN may interact in a more complex and interdependent way than in a conventional network, appropriate testing
methodologies are required in all phases of operation. For instance, the effect of software updates in network
components can be amplified by the more connected nature of these components in an AN.
The rest of the present document is organized as follows:
• Clause 4 gives the definition of an adaptive network, as used in the context of the present document.
• Clause 5 defines the entities and interactions that may be encountered in an adaptive network.
• Clause 6 defines the general functional targets that should be met by adaptive networks.
• Clause 7 defines the methods that may be used to test adaptive networks.
ETSI
6 ETSI EG 203 341 V1.1.1 (2016-10)
• Annex A gives an overview of the relation of the present document to other work performed in this area,
e.g. NFV TST, NTECH-AFI.
ETSI
7 ETSI EG 203 341 V1.1.1 (2016-10)
1 Scope
The present document, "Approaches for Testing Adaptive Networks" defines a framework of testing principles and
guidelines that may be used to test networks that exhibit some form of autonomic adaptive behaviour, which allows
them to dynamically change their configuration, structure or operational parameters. The (re)-configuration is
performed in response to stimuli such as changes in workload, operator policies that govern their operation, context (the
network is context-aware and may have a degree of self-awareness); and challenges in the environment (i.e. conditions
under which the network is operating, e.g. manifestations of faults, errors, failures in various parts of the network and
its hardware and software components).
The functionality of individual components and basic interoperability can be ensured at design time. However, the
complex interactions between various components or functions deployed in a live Adaptive Network (AN) may not be
fully assessed or foreseen. Consequently, the document addresses methodologies to test ANs towards meeting their
functional targets or policies, and ensuring a minimum trust level for autonomic operation of such networks.
NOTE: In the literature, both the terms "autonomous" and "autonomic" are being used in this context, whereas
"autonomous" appears to indicate a higher level of automation. As adaptive networks are, at the time of
writing, surely a technology still at its beginnings, "autonomic" may be a less ambitious and therefore
more appropriate term for the time being. On the other hand, the NGMN 5G White Paper (V1.0) uses the
term combination "autonomic/self-management functions" which points, clearly towards a level beyond
"autonomic". As mobile networks are complex systems, it is most likely that the degree of automation
will increase in the course of technical evolution, but not in an isotropic way; there will be areas with
higher and others with lower levels of automation, and sophistication of respective functions. For these
reasons, the present document will use the term "autonomic".
2 References
2.1 Normative references
Normative references are not applicable in the present document.
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] ETSI GS AFI 002: "Autonomic network engineering for the self-managing Future Internet (AFI);
Generic Autonomic Network Architecture (An Architectural Reference Model for Autonomic
Networking, Cognitive Networking and Self-Management)".
[i.2] ETSI TS 102 250-4: "Speech and multimedia Transmission Quality (STQ); QoS aspects for
popular services in mobile networks; Part 4: Requirements for Quality of Service measurement
equipment".
[i.3] Recommendation ITU-T P.10/G.100 Amendment 2 (07/2008): "Vocabulary for performance and
quality of service Amendment 2: New definitions for inclusion in Recommendation ITU-T
P.10/G.100".
[i.4] Recommendation ITU-T E.800 (09/2008): "Definitions of terms related to quality of service".
[i.5] ISO/IEC 9646: "Information technology - Open Systems Interconnection - Conformance testing
methodology and framework".
ETSI
8 ETSI EG 203 341 V1.1.1 (2016-10)
[i.6] ETSI GS NFV-TST 001 (V1.1.1): "Network Functions Virtualisation (NFV); Pre-deployment
Testing; Report on Validation of NFV Environments and Services".
[i.7] ETSI GS NFV-TST 002: "Network Functions Virtualisation (NFV); Testing Methodology; Report
on Interoperability Testing Methodology".
[i.8] Dar, K.: "Autonomic Computing: An introduction to MAPE-K reference model".
NOTE: Available at http://www.uio.no/studier/emner/matnat/ifi/INF5360/v13/undervisningsmateriale/mape-
k.pdf.
[i.9] IBM (2005):"An architectural blueprint for autonomic computing".
NOTE: Available at http://www-03.ibm.com/autonomic/pdfs/AC%20Blueprint%20White%20Paper%20V7.pdf.
[i.10] Hayan, Z.: "A novel autonomic architecture for QoS management in wired network".
NOTE: Available at
http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=5700376&url=http%3A%2F%2Fieeexplore.ieee.o
rg%2Fxpls%2Fabs_all.jsp%3Farnumber%3D5700376).
[i.11] Strassner, J., Agoulmine, N., & Lethihet, E. (2006): "FOCALE - A Novel Autonomic Networking
Architecture".
NOTE: Available at http://repository.wit.ie/189/1/2006_LAACS_Strassner_et_al_final.pdf.
[i.12] Clark, D. C., Partridge, C., Ramming, J. C., Wroclawski, J. T.: "A knowledge plane for the
internet".
3 Definitions and abbreviations
3.1 Definitions
For the purposes of the present document, the following terms and definitions apply:
aggregation hierarchy: description of how detailed (granular) performance data will be aggregated into summary data,
and vice versa, how to break down the summary data into details
attractor: state or behaviour toward which a dynamic system tends to evolve, represented as a point or orbit in the
system's phase space
control loop: mechanism which uses observations of a system to make modifications to the observed system to meet a
given target
3.2 Abbreviations
For the purposes of the present document, the following abbreviations apply:
AF Adaptive Function
AFI Autonomic Future Internet
AMC Autonomic Management and Control
AN Adaptive Network
CCO Coverage and Capacity Optimization
DE Decision Element
eNB evolved Node B
FUT Function Under Test
GANA Generic Autonomic Network Architecture
IBM International Business Machines
ISG Industry Specification Group
ITU-T International Telecommunication Union - Telecommunication standardization sector
KPI Key Performance Indicator
ETSI
9 ETSI EG 203 341 V1.1.1 (2016-10)
LTE Long-Term Evolution
MEC Mobile Edge Computing
MRO Mobility Robustness Optimization
NE Network Element
NFV Network Functions Virtualisation
NGMN Next Generation Mobile Networks
NTECH Network Technologies
NUT Network Under Test
OCS Overall Configuration State (of a network)
ONP Overall Network Properties
QoE Quality of Experience
QoS Quality of Service
RAN Radio Access Network
SDN Software Defined Networking
SLA Service Level Agreement
SON Self Organizing Networks
UE User Equipment
VoLTE Voice over LTE
4 Definition of Adaptive Networks
4.1 Basic Concept
The term "adaptive network" (AN) refers to any network that has the ability to automatically modify its configuration,
operational parameters or structure, in order to comply with pre-defined functional targets or operational policies, and
with the ability to handle situations that were unknown at its design time (e.g. with predictions and forecasting
capabilities as well), thus producing a dynamic environment with multiple potential network states. An adaptive
network may include technologies such as Self Organizing Networks (SON), Network Functions Virtualisation (NFV),
Software Defined Networking (SDN), Autonomic Management and Control (AMC) or any other technology which
enables a network to exhibit the characteristics mentioned above.
Adaptive networks are comprised of one or more Adaptive Functions (AF) that dynamically and adaptively manage and
control certain network attributes. These functions are fundamentally characterized by exhibiting control-loops which
can be embedded at different layers e.g. protocol level, node level, network level, and exert different degrees of
influence over the network. Similarly, the management and control of the AFs can be aggregated at different levels
depending on the information required for their operation. Furthermore, ANs may function on different time scales and
with different levels of complexity and views on which they operate on, depending on the type of AFs that are
deployed. However, from an end user perspective, the presence or absence of AFs in a network is transparent, meaning
that end users can only observe the functionality of the network service. Similarly to conventional networks, the internal
structure and operation of the network is not visible from this perspective.
Depending on the type of AFs and the level where they are deployed, the frequency of changes performed throughout
an AN can differ. In general, low level AFs can operate at faster time scales, i.e. fast control loops as they utilize
information collected locally. On the other hand, high level AFs require information about the overall state of the
network and thus typically operate in slow control loops. The architecture of an AN, in terms of the hierarchical
placement of AFs and aggregation levels is important from a testing perspective and determines if and how the
particular network can be tested. Figure 1 illustrates the different architectures of ANs and the associated control loops.
Two extreme cases can be distinguished:
• Fully distributed adaptive network, where all AFs operate at lower levels, e.g. at the protocol or node level,
with no management and control aggregation at higher levels.
• Fully centralized adaptive network, where AFs operate at higher levels, e.g. network level and aggregate
network wide information.
ETSI
10 ETSI EG 203 341 V1.1.1 (2016-10)
The fully distributed architecture poses higher challenges from a testing perspective, since the effect of AFs that operate
in fast control loops may not be easily translated into functional KPIs that can be observed by a test system.
Furthermore, their policies and functional targets are managed and executed locally, at an aggregation level where
information may not be available for a test system. On the other hand, the fully centralized architecture is the most
attractive from a testing perspective, since it operates using slow control loops and uses information that is aggregated
at network level.
A typical AN will incorporate several types of AFs, that operate and aggregate information at different levels. Hence,
from an architectural perspective it may use a hybrid model, which includes distributed, and centralized AFs or AFs that
are aggregated at an intermediated level. Additionally, a peer-to-peer relationship may be formed between AFs
operating at the same hierarchical level.
Management and Management and Management and
Policy Policy Policy
AF AF
AF
AF
NE
AF
AF NE NE
AF
AF
NE
NE
NE NE
NE NE
AF
AF
NE
NE
NE
Figure 1: Adaptive Network Architectures: distributed, centralized and hybrid
The detailed internal structure and algorithms of the AN may not be known to an external test environment. However, a
minimum set of information regarding the operation and structure of the AN may be required in order to interpret results
generated from end-to-end functionality testing. This information can include details about the functional targets of the
AN, the capabilities of AFs that are deployed, their operational status, e.g. active, idle, disabled, the network attributes
that they control and their influence on the functional target being measured. Part of the information may be obtained
out of band, i.e. be provided as external input to the test system, while part of the information may be obtained from the
Network Under Test (NUT).
An adaptive network typically functions in a closed loop manner, with minimum human intervention using sensor
information to make decisions and perform actions, according to policies set by the network operator. These actions can
be categorized in:
• Actions that are performed on network configuration parameters or network resources, e.g. Transmission
Power, antenna tilt, routing policies, bandwidth allocation.
• Actions that are performed on the network structure, e.g. adding/removing network elements (either physical
or virtualized instances). These actions imply configuration changes in order to accommodate the structural
change.
The events that can trigger an adaptive network to dynamically change its properties vary also depending on the specific
AFs deployed in the network and the level at which they operate. They can be split in two categories:
• Externally generated events - when the adaptive behaviour is triggered by an external factor, e.g. increase in
user traffic that creates unbalanced load in the network, detecting service-level performance degradation,
failure of network elements.
• Internally generated events - when the adaptive behaviour is triggered as a result of an internal policy,
independent of external activity, e.g. power savings mode, configuration of network properties to provide QoS
for certain traffic types, e.g. low latency traffic, delay-tolerant traffic, low-bandwidth traffic.
NOTE: These events can occur in a chain like fashion, e.g. policy change can trigger several secondary events in
lower level functional units.
ETSI
11 ETSI EG 203 341 V1.1.1 (2016-10)
4.2 General Terminology
4.2.1 Introduction
A fundamental characteristic of ANs is the ability to dynamically change their configuration and properties. In order to
describe the testing methodology some basic concepts (configuration states, state transitions and attractors) have to be
introduced, as their meaning is new or goes beyond well-known definitions for conventional networks.
4.2.2 Network States
A network is characterized by its hardware and software components, together with the configuration of these
components. This configuration is given by control elements, which can be on hardware level (e.g. elements
determining physical orientation of antennas) or on software level (parameters determining the functional behaviour of
a component). A component can have multiple control elements which define its overall state. Similarly, the overall
network state is defined by the overall states of each component. The total number of these controls - counting each
degree of freedom separately - is typically large, but finite and a fixed property of a given network.
Each degree of freedom can be:
• a discrete value, out of a given set of choices or a range of integer values; or
• a continuous (analogue) value.
The totality of all degrees of freedom represents the settings space. Each combination of settings can be described as an
N-dimensional vector, where N is the number of degrees of freedom, also called the dimension of the settings space. An
individual control setting is then the i-th element of this settings vector.
Each possible combination of settings is represented by the corresponding vector. For the purpose of the present
document, such a vector is termed Overall Configuration State (OCS).
A change of settings - regardless if done by human operators as in conventional networks or by automatic processes in
AN - means a transition between an initial OCS S to a new OCS S .
1 2
Resource A Configuration 1 Resource A Configuration 2
Control A_a: {a1} Control A_a: {a5}
C C
o o
n n
t t
r r
o o
l l
A A
Resource Resource
_ _
b b
A A
: :
{ {
b b
7 7
} }
Control A_c: {c3}
Control A_c: {c1}
Resource Resource Resource Resource Resource Resource
A B N A B N
{}. { .
OCS S OCS S
1 2
Figure 2: Concept of controls and Overall Configuration State (OCS) transitions
Also for this purpose and later usage, the term overall network properties (ONP) is defined which describes the
appearance of the network as perceivable from the end user point of view or through other interfaces to the network
operator (see also clause 5.1). Each OCS leads to a specific ONP.
NOTE: This relation is not symmetric; several OCS can lead to the same ONP, but the assumption is that the
same OCS cannot lead to different ONP. If this was the case it would mean that some aspect of the
network shows random behaviour which is a primarily unwanted condition.
ETSI
12 ETSI EG 203 341 V1.1.1 (2016-10)
4.2.3 Static and stationary states
In a conventional network, where controls are operated by humans, it is likely that settings, once made, do not
frequently change after they have been made. In contrast, in ANs, settings and associated network properties constantly
change as a result of various AF that operate in the network.
NOTE 1: In the context of the present clause, the term "state" represents the OCS, as introduced in clause 4.2.2.
S1
st atic
S2
stationary
Figure 3: Explanation of static and stationary states
To describe this situation, figure 3 shows a two-dimensional state space with two entities, S1 and S2.
S1 and S2 represent two types of states. A state which is constant over time (S1) is called static. A state which
fluctuates over time, around an identifiable point in the state space is called stationary (S2).Independent of the actual
shape or distribution of values, the essential property of a stationary state is that fluctuations occur within a given area,
which is sufficiently small compared to the overall state space.
NOTE 2: The definition of "small" is of course somewhat arbitrary. A pragmatic definition may be that effects on
the ONP are small against measurement errors in determining these properties.
The time scale of fluctuations is also an important characteristic of a stationary state. It will depend on both the
properties and capabilities of respective control elements and the characteristics of the decision processes in operation.
For later reference, state changes are called "microscopic" if they do not have a practical effect on ONP, and
"macroscopic" if they do.
In this state picture, instability either means large cyclical or chaotic fluctuations of the OCS with observable effects on
ONP, or a network state which is pulled towards some state with unwanted (unusable) ONP. Clearly, to determine the
temporal behaviour requires time which is - in addition to statistical reasons on sample number - the reason why such
measurements need appropriate time spans to perform.
4.2.4 State Transitions and Attractors
After having introduced the concept of static and stationary states, the question is how a NUT might change its state in
the course of the adaptation process. For illustration see figure 4.
ETSI
13 ETSI EG 203 341 V1.1.1 (2016-10)
S1
st atic
S2
st ationary
Figure 4: Examples of state transition paths
Here, S1 is the state before the adaptation starts (initial state), and S2 is the state after the adaptive process is completed
(end state). The figure shows two paths from S1 to S2, a direct one and an indirect one. The actual path depends on the
adaptive algorithms being used. Even if it appears unlikely that a NUT actually shows a behaviour as the one shown in
this example, it cannot be excluded either. It may be the consequence of restrictions in network resource control or of
actual properties of the algorithms used. Also, it is conceivable that such a behaviour is, in distributed adaptive
networks, the result of interplay between "local" actions.
From the association between internal network configuration (OCS) and network performance (ONP), it follows that
during a state transition, the QoS of the network may be degraded. From the testing viewpoint, this has to be considered
too. While such a temporary degradation may be unavoidable in general, the impact as seen from a network subscriber's
perspective will depend on its duration and seriousness. Therefore, respective properties need to be considered in the
functional targets and assessment procedures used in testing. For example - in case of comparative testing or
benchmarking of two ANs, one candidate may exhibit a faster adaptation process, or an adaptation towards a better end
state while exhibiting a more serious or longer period of degradation than the other.
Basically, a network can have any state physically or technically possible, i.e. the initial state can be any point in state
space. If an adaptive process sets in, the state will - if the NUT is not unstable - move towards the end state. As the
adaptive process is actually an optimization of network parameters, there will be a finite number of end states, each of
which represents a local optimum, or the global optimum of the network with respect to the targets given by
administrative policies and current operating conditions. Figure 5 shows an example.
ETSI
14 ETSI EG 203 341 V1.1.1 (2016-10)

Figure 5: State space with local optima shown as end states
The points represent examples for initial states. If the NUT is brought into one of these states - and if such a state does
not already represent a stable or stationary state - and adaptation is enabled, the adaptation will lead, according to the
adaptation algorithm, to an end state. The concept of optimization implies that most of the possible states are not
optimal with respect to the rule set applied. Therefore, a transition from the initial state to a "better" state will occur.
Under the assumption that there are multiple (local) optima, the state space will have regions of initial states which lead
to different (optimized) end states.
NOTE: The state space may have characteristics which do not allow a direct path from a given starting point to
the global optimum of the system, by applying an incremental (mathematical, e.g. gradient based)
optimization algorithm.
Initial states in the left region lead to the end state E1; initial states in the right region lead to end state E2. In analogy to
the use of this term in other areas E1 and E2 are called attractors of the system.
The shape of the attractor landscape may have considerable effect on the dynamic behaviour and the predictability of
the system in the field. Assume a situation with a complex or rather fragmented attractor space. Two starting points
which are close to each other in the state space, may have attractors associated with quite different network
configurations. In a laboratory environment the degree of control over the starting points is higher compared to
operational networks. In effect, this may limit the ability to predict which configurations will be reached in actual
operation.
The situations described above are idealized by assuming that during the time an adaptive process is taking place, the
conditions which had caused this adaptation remain constant. If conditions change during this transition, and
considering a fragmented attractor space, there may be a high probability that the system is oscillating between end
states with probable negative effects on QoS. The test strategy should define means to detect such situations.
4.3 Adaptive Networks as Network Under Test
Testing ANs, implies testing a system of AFs that operate towards meeting functional targets defined by the network
policies. The scenario can be compared to traditional interoperability testing, where the goal is to verify the end-to-end
functionality (as experienced by a user) of several Functions Under Test (FUT).
Individual AFs typically pass through a conformance testing procedure at design time. However, AFs may be coupled
and interact during operation, leading to situations that were not anticipated beforehand. Consequently, testing
individual (or subsets) of AFs may not guarantee proper end-to-end functionality of the AN, unless it can be ensured
that the tested function (or subset) is independent from other AFs that operate in the same AN.
ETSI
15 ETSI EG 203 341 V1.1.1 (2016-10)
AF1 AF3
AF4
AF2
Figure 6: Example of NUT comprised of several AFs and potential interactions
The complexity of the NUT is given by the number of AFs it consists of, the hierarchical level at which they operate
and are aggregated and also the time scales at which they operate. AFs may be standalone or interconnected with other
AFs.
For testing purposes, it may be helpful to split a complex NUT into smaller segments. However, it is essential that any
split does not impact control loops, in order to avoid altering the dynamic behaviour of the NUT. Potential criteria that
may be used to segment an NUT are:
• Hierarchical aggregation level - the adaptive NUT will be tested only at a specific aggregation level.
• Time scale - the adaptive NUT will be tested only for adaptive functions that operate on a certain time scale,
e.g. slow control loops.
• Functional target - the adaptive NUT will be tested only towards a certain number of functional targets.
However, as discussed above, testing different segments of the NUT may not be equivalent to testing the NUT from an
end-to-end perspective.
5 Entities and interactions
5.1 Overview
Figure 7 shows the principal testing environment for adaptive network testing. It consists of:
• the network under test (NUT);
• effectors which constitute the stimuli for testing;
• sensors which provide information about the NUT;
• optional monitoring points for internal NUT information.
ETSI
16 ETSI EG 203 341 V1.1.1 (2016-10)
Monitoring Information
S
A1
R
O
T
C A2
E
F
F
E
NUT
S I1
R
O
S
I2
N
E
S
Figure 7: Testing environment
The test control domain is not shown in figure 7. The NUT is treated as a solid "black box" from a dynamic
(behavioural) point of view; see clause 4 for assumptions about its inner structure. With respect to the (logical)
architecture, assumption is that it can be described, on an abstract level, by a generic model, e.g. GANA. Given the
nature and current state of development of ANs, any further assumptions about architectural or structural details should
be avoided as these may be misleading.
On the effector side, activity type A1 denotes activities which are equivalent or identical to those coming, in real
network operation, from end users. They include all types of traffic that can be applied to the network, e.g. audio or
video calls (e.g. VoLTE or legacy telephony) as well as usage of data services such as Web Browsing, video streaming
and other types of packet data based activities. A1 activities may include also any form of machine-type traffic relevant
to the tested network.
Activity type A2 is the category for actions towards the NUT which cannot be triggered by end users. They include
structural actions such as addition or removal of physical or virtualized network elements. Also, A2 activities include
policy modifications or changes in defined functional targets that cause adaptive functions to change network settings or
behaviour.
Likewise, on the sensor side two general types of information are distinguished:
• Sensor information type I1 is information related to properties of the network which are visible to subscribers,
i.e. QoS information such as accessibility, retainability, throughput, or latency.
• Sensor information type I2 is information which is not directly visible to subscribers but may have an
informational or business value for the operator, e.g. energy consumption of network nodes, traffic load or
traffic mix.
I1 and I2 can, be described as multi-dimensional vectors (where the number of dimensions is network specific). One
specific instance, therefore represents the overall network properties (ONP) as introduced in clause 4.2.2, at the
respective point in time.
With respect to elaborations made in previous clauses, information of type I2 may require additional interfaces or
insight into the network's structure or operation which cannot be assumed to exist in general.
The additional output labelled "Monitoring Information" stands for detailed information on the NUT's internal state and
dynamics. The availability of such information is not mandatory, for the testing methodologies described in the present
document. However, such additional information, beyond the information required to test the NUT from a functional
point of view can be useful for diagnostics.
ETSI
17 ETSI EG 203 341 V1.1.1 (2016-10)
5.2 Effectors/Activities
5.2.1 User-equivalent activities (type A1)
5.2.1.1 Introduction
User equivalent activities comprise all types of traffic that can be generated by an end-user of the network, e.g. traffic
scenarios, auto-configuration of network devices, procedures between device and network. Typical means to create such
scenarios are load, QoS testing systems. From a scenario point of vie
...

SLOVENSKI STANDARD
01-november-2016
-HGUQRRPUHåMHLQSUHVNXãDQMHPHGREUDWRYDOQRVWL,171DþLQLSUHVNXãDQMD
SULODJRGOMLYLKRPUHåLM
Core Network and Interoperability Testing (INT) - Approaches for Testing Adaptive
Networks
Ta slovenski standard je istoveten z: ETSI EG 203 341 V1.1.1 (2016-08)
ICS:
33.040.01 Telekomunikacijski sistemi Telecommunication systems
na splošno in general
2003-01.Slovenski inštitut za standardizacijo. Razmnoževanje celote ali delov tega standarda ni dovoljeno.

Final draft ETSI EG 203 341 V1.1.1 (2016-08)

ETSI GUIDE
Core Network and Interoperability Testing (INT);
Approaches for Testing Adaptive Networks

2 Final draft ETSI EG 203 341 V1.1.1 (2016-08)

Reference
DEG/INT-00127
Keywords
conformance, interoperability, methodology
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: +33 4 92 94 42 00  Fax: +33 4 93 65 47 16

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
The present document can be downloaded from:
http://www.etsi.org/standards-search
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the only prevailing document is the
print of the Portable Document Format (PDF) version kept on a specific network drive within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
https://portal.etsi.org/TB/ETSIDeliverableStatus.aspx
If you find errors in the present document, please send your comment to one of the following services:
https://portal.etsi.org/People/CommiteeSupportStaff.aspx
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© European Telecommunications Standards Institute 2016.
All rights reserved.
TM TM TM
DECT , PLUGTESTS , UMTS and the ETSI logo are Trade Marks of ETSI registered for the benefit of its Members.
TM
3GPP and LTE™ are Trade Marks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.
ETSI
3 Final draft ETSI EG 203 341 V1.1.1 (2016-08)
Contents
Intellectual Property Rights . 5
Foreword . 5
Modal verbs terminology . 5
Introduction . 5
1 Scope . 7
2 References . 7
2.1 Normative references . 7
2.2 Informative references . 7
3 Definitions and abbreviations . 8
3.1 Definitions . 8
3.2 Abbreviations . 8
4 Definition of Adaptive Networks . 9
4.1 Basic Concept . 9
4.2 General Terminology . 11
4.2.1 Introduction. 11
4.2.2 Network States . 11
4.2.3 Static and stationary states . 12
4.2.4 State Transitions and Attractors . 12
4.3 Adaptive Networks as Network Under Test . 14
5 Entities and interactions . 15
5.1 Overview . 15
5.2 Effectors/Activitie s . 17
5.2.1 User-equivalent activities (type A1) . 17
5.2.1.1 Introduction . 17
5.2.1.2 Systems delivering the required functionality . 17
5.2.2 Structural or other activities (type A2) . 17
5.2.2.1 Introduction . 17
5.2.2.2 Systems delivering the required functionality . 17
5.2.3 Additional controls . 18
5.3 Information/Sensors . 18
5.3.1 Network performance from end user perspective (type I1) . 18
5.3.1.1 Introduction . 18
5.3.1.2 Systems delivering the required functionality . 18
5.3.2 Additional information about the network (type I2) . 19
5.3.3 Additional aspects of sensors . 19
6 Functional Targets . 19
6.1 Introduction . 19
6.2 Network stages . 19
6.3 Classes of functional targets . 21
6.4 Applicability of functional targets to network stages . 22
7 Generic Framework and Methods for Testing Adaptive Networks . 22
7.1 Basic Assumptions . 22
7.2 General aspects and related terminology . 23
7.3 Testing Process . 23
7.3.1 Introduction. 23
7.3.2 A1 based testing scenarios . 23
7.3.3 A2 based testing scenarios . 24
7.4 Evaluation of results . 25
Annex A: Relation to other work done in this field. 26
A.1 Introduction . 26
ETSI
4 Final draft ETSI EG 203 341 V1.1.1 (2016-08)
A.2 ISG NFV . 26
A.2.1 Group description . 26
A.2.2 Network Functions Virtualisation (NFV) . 26
A.3 NTECH AFI . 26
A.3.1 Group description . 26
A.3.2 GANA model overview . 27
A.3.3 Concepts of the Generic Test Framework for Testing Adaptive Functions . 27
History . 31

ETSI
5 Final draft ETSI EG 203 341 V1.1.1 (2016-08)
Intellectual Property Rights
IPRs essential or potentially essential to the present document may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (https://ipr.etsi.org/).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Foreword
This final draft ETSI Guide (EG) has been produced by ETSI Technical Committee Core Network and Interoperability
Testing (INT), and is now submitted for the ETSI standards Membership Approval Procedure.
Modal verbs terminology
In the present document "should", "should not", "may", "need not", "will", "will not", "can" and "cannot" are to be
interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
Introduction
The characteristics of "adaptive networks" such as virtualization, self-organization, self-configuration, self-
optimization, self-healing and self-learning, dynamic network slicing promise to offer huge advantages in future
networks. While technologies such as Network Functions Virtualisation (NFV), Self-Organizing Networks (SON),
Mobile Edge Computing (MEC) and Autonomic Management and Control (AMC) of Networks and Services may not
each exhibit all the characteristics they do have one thing in common: they are all dynamic rather than static, reacting to
dynamic traffic conditions, applications, service demands as well as to changes in the eco-system environment.
By incorporating one or several of the technologies mentioned above, Adaptive Networks (AN) have the ability to
automatically and dynamically manage and control network resources, configuration parameters or the network
structure, with limited human intervention, in order to meet functional targets or operational policies. However, to
achieve this type of autonomic behaviour, it has to be ensured that any modification that is performed automatically in
the network does not produce undesired effects, e.g. instability or lower performance with respect to the end-user
perspective.
Comprehensive testing, both on a general level as in type approvals and related to acceptance testing of a particular
deployment, is therefore even more important than it is for conventional networks. Due to the fact that the components
of an AN may interact in a more complex and interdependent way than in a conventional network, appropriate testing
methodologies are required in all phases of operation. For instance, the effect of software updates in network
components can be amplified by the more connected nature of these components in an AN.
The rest of the present document is organized as follows:
• Clause 4 gives the definition of an adaptive network, as used in the context of the present document.
• Clause 5 defines the entities and interactions that may be encountered in an adaptive network.
• Clause 6 defines the general functional targets that should be met by adaptive networks.
• Clause 7 defines the methods that may be used to test adaptive networks.
ETSI
6 Final draft ETSI EG 203 341 V1.1.1 (2016-08)
• Annex A gives an overview of the relation of the present document to other work performed in this area,
e.g. NFV TST, NTECH-AFI.
ETSI
7 Final draft ETSI EG 203 341 V1.1.1 (2016-08)
1 Scope
The present document, "Approaches for Testing Adaptive Networks" defines a framework of testing principles and
guidelines that may be used to test networks that exhibit some form of autonomic adaptive behaviour, which allows
them to dynamically change their configuration, structure or operational parameters. The (re)-configuration is
performed in response to stimuli such as changes in workload, operator policies that govern their operation, context (the
network is context-aware and may have a degree of self-awareness); and challenges in the environment (i.e. conditions
under which the network is operating, e.g. manifestations of faults, errors, failures in various parts of the network and
its hardware and software components).
The functionality of individual components and basic interoperability can be ensured at design time. However, the
complex interactions between various components or functions deployed in a live Adaptive Network (AN) may not be
fully assessed or foreseen. Consequently, the document addresses methodologies to test ANs towards meeting their
functional targets or policies, and ensuring a minimum trust level for autonomic operation of such networks.
NOTE: In the literature, both the terms "autonomous" and "autonomic" are being used in this context, whereas
"autonomous" appears to indicate a higher level of automation. As adaptive networks are, at the time of
writing, surely a technology still at its beginnings, "autonomic" may be a less ambitious and therefore
more appropriate term for the time being. On the other hand, the NGMN 5G White Paper (V1.0) uses the
term combination "autonomic/self-management functions" which points, clearly towards a level beyond
"autonomic". As mobile networks are complex systems, it is most likely that the degree of automation
will increase in the course of technical evolution, but not in an isotropic way; there will be areas with
higher and others with lower levels of automation, and sophistication of respective functions. For these
reasons, the present document will use the term "autonomic".
2 References
2.1 Normative references
Normative references are not applicable in the present document.
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] ETSI GS AFI 002: "Autonomic network engineering for the self-managing Future Internet (AFI);
Generic Autonomic Network Architecture (An Architectural Reference Model for Autonomic
Networking, Cognitive Networking and Self-Management)".
[i.2] ETSI TS 102 250-4: "Speech and multimedia Transmission Quality (STQ); QoS aspects for
popular services in mobile networks; Part 4: Requirements for Quality of Service measurement
equipment".
[i.3] Recommendation ITU-T P.10/G.100 Amendment 2 (07/2008): "Vocabulary for performance and
quality of service Amendment 2: New definitions for inclusion in Recommendation ITU-T
P.10/G.100".
[i.4] Recommendation ITU-T E.800 (09/2008): "Definitions of terms related to quality of service".
[i.5] ISO/IEC 9646: "Information technology - Open Systems Interconnection - Conformance testing
methodology and framework".
ETSI
8 Final draft ETSI EG 203 341 V1.1.1 (2016-08)
[i.6] ETSI GS NFV-TST 001 (V1.1.1): "Network Functions Virtualisation (NFV); Pre-deployment
Testing; Report on Validation of NFV Environments and Services".
[i.7] ETSI GS NFV-TST 002: "Network Functions Virtualisation (NFV); Testing Methodology; Report
on Interoperability Testing Methodology".
[i.8] Dar, K.: "Autonomic Computing: An introduction to MAPE-K reference model".
NOTE: Available at http://www.uio.no/studier/emner/matnat/ifi/INF5360/v13/undervisningsmateriale/mape-
k.pdf.
[i.9] IBM (2005):"An architectural blueprint for autonomic computing".
NOTE: Available at http://www-03.ibm.com/autonomic/pdfs/AC%20Blueprint%20White%20Paper%20V7.pdf.
[i.10] Hayan, Z.: "A novel autonomic architecture for QoS management in wired network".
NOTE: Available at
http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=5700376&url=http%3A%2F%2Fieeexplore.ieee.o
rg%2Fxpls%2Fabs_all.jsp%3Farnumber%3D5700376).
[i.11] Strassner, J., Agoulmine, N., & Lethihet, E. (2006): "FOCALE - A Novel Autonomic Networking
Architecture".
NOTE: Available at http://repository.wit.ie/189/1/2006_LAACS_Strassner_et_al_final.pdf.
[i.12] Clark, D. C., Partridge, C., Ramming, J. C., Wroclawski, J. T.: "A knowledge plane for the
internet".
3 Definitions and abbreviations
3.1 Definitions
For the purposes of the present document, the following terms and definitions apply:
aggregation hierarchy: description of how detailed (granular) performance data will be aggregated into summary data,
and vice versa, how to break down the summary data into details
attractor: state or behaviour toward which a dynamic system tends to evolve, represented as a point or orbit in the
system's phase space
control loop: mechanism which uses observations of a system to make modifications to the observed system to meet a
given target
3.2 Abbreviations
For the purposes of the present document, the following abbreviations apply:
AF Adaptive Function
AFI Autonomic Future Internet
AMC Autonomic Management and Control
AN Adaptive Network
CCO Coverage and Capacity Optimization
DE Decision Element
eNB evolved Node B
FUT Function Under Test
GANA Generic Autonomic Network Architecture
IBM International Business Machines
ISG Industry Specification Group
ITU-T International Telecommunication Union - Telecommunication standardization sector
KPI Key Performance Indicator
ETSI
9 Final draft ETSI EG 203 341 V1.1.1 (2016-08)
LTE Long-Term Evolution
MEC Mobile Edge Computing
MRO Mobility Robustness Optimization
NE Network Element
NFV Network Functions Virtualisation
NGMN Next Generation Mobile Networks
NTECH Network Technologies
NUT Network Under Test
OCS Overall Configuration State (of a network)
ONP Overall Network Properties
QoE Quality of Experience
QoS Quality of Service
RAN Radio Access Network
SDN Software Defined Networking
SLA Service Level Agreement
SON Self Organizing Networks
UE User Equipment
VoLTE Voice over LTE
4 Definition of Adaptive Networks
4.1 Basic Concept
The term "adaptive network" (AN) refers to any network that has the ability to automatically modify its configuration,
operational parameters or structure, in order to comply with pre-defined functional targets or operational policies, and
with the ability to handle situations that were unknown at its design time (e.g. with predictions and forecasting
capabilities as well), thus producing a dynamic environment with multiple potential network states. An adaptive
network may include technologies such as Self Organizing Networks (SON), Network Functions Virtualisation (NFV),
Software Defined Networking (SDN), Autonomic Management and Control (AMC) or any other technology which
enables a network to exhibit the characteristics mentioned above.
Adaptive networks are comprised of one or more Adaptive Functions (AF) that dynamically and adaptively manage and
control certain network attributes. These functions are fundamentally characterized by exhibiting control-loops which
can be embedded at different layers e.g. protocol level, node level, network level, and exert different degrees of
influence over the network. Similarly, the management and control of the AFs can be aggregated at different levels
depending on the information required for their operation. Furthermore, ANs may function on different time scales and
with different levels of complexity and views on which they operate on, depending on the type of AFs that are
deployed. However, from an end user perspective, the presence or absence of AFs in a network is transparent, meaning
that end users can only observe the functionality of the network service. Similarly to conventional networks, the internal
structure and operation of the network is not visible from this perspective.
Depending on the type of AFs and the level where they are deployed, the frequency of changes performed throughout
an AN can differ. In general, low level AFs can operate at faster time scales, i.e. fast control loops as they utilize
information collected locally. On the other hand, high level AFs require information about the overall state of the
network and thus typically operate in slow control loops. The architecture of an AN, in terms of the hierarchical
placement of AFs and aggregation levels is important from a testing perspective and determines if and how the
particular network can be tested. Figure 1 illustrates the different architectures of ANs and the associated control loops.
Two extreme cases can be distinguished:
• Fully distributed adaptive network, where all AFs operate at lower levels, e.g. at the protocol or node level,
with no management and control aggregation at higher levels.
• Fully centralized adaptive network, where AFs operate at higher levels, e.g. network level and aggregate
network wide information.
ETSI
10 Final draft ETSI EG 203 341 V1.1.1 (2016-08)
The fully distributed architecture poses higher challenges from a testing perspective, since the effect of AFs that operate
in fast control loops may not be easily translated into functional KPIs that can be observed by a test system.
Furthermore, their policies and functional targets are managed and executed locally, at an aggregation level where
information may not be available for a test system. On the other hand, the fully centralized architecture is the most
attractive from a testing perspective, since it operates using slow control loops and uses information that is aggregated
at network level.
A typical AN will incorporate several types of AFs, that operate and aggregate information at different levels. Hence,
from an architectural perspective it may use a hybrid model, which includes distributed, and centralized AFs or AFs that
are aggregated at an intermediated level. Additionally, a peer-to-peer relationship may be formed between AFs
operating at the same hierarchical level.
Management and Management and Management and
Policy Policy Policy
AF AF
AF
AF
NE
AF
AF NE NE
AF
AF
NE
NE NE NE
NE NE
AF
AF
NE
NE
NE
Figure 1: Adaptive Network Architectures: distributed, centralized and hybrid
The detailed internal structure and algorithms of the AN may not be known to an external test environment. However, a
minimum set of information regarding the operation and structure of the AN may be required in order to interpret results
generated from end-to-end functionality testing. This information can include details about the functional targets of the
AN, the capabilities of AFs that are deployed, their operational status, e.g. active, idle, disabled, the network attributes
that they control and their influence on the functional target being measured. Part of the information may be obtained
out of band, i.e. be provided as external input to the test system, while part of the information may be obtained from the
Network Under Test (NUT).
An adaptive network typically functions in a closed loop manner, with minimum human intervention using sensor
information to make decisions and perform actions, according to policies set by the network operator. These actions can
be categorized in:
• Actions that are performed on network configuration parameters or network resources, e.g. Transmission
Power, antenna tilt, routing policies, bandwidth allocation.
• Actions that are performed on the network structure, e.g. adding/removing network elements (either physical
or virtualized instances). These actions imply configuration changes in order to accommodate the structural
change.
The events that can trigger an adaptive network to dynamically change its properties vary also depending on the specific
AFs deployed in the network and the level at which they operate. They can be split in two categories:
• Externally generated events - when the adaptive behaviour is triggered by an external factor, e.g. increase in
user traffic that creates unbalanced load in the network, detecting service-level performance degradation,
failure of network elements.
• Internally generated events - when the adaptive behaviour is triggered as a result of an internal policy,
independent of external activity, e.g. power savings mode, configuration of network properties to provide QoS
for certain traffic types, e.g. low latency traffic, delay-tolerant traffic, low-bandwidth traffic.
NOTE: These events can occur in a chain like fashion, e.g. policy change can trigger several secondary events in
lower level functional units.
ETSI
11 Final draft ETSI EG 203 341 V1.1.1 (2016-08)
4.2 General Terminology
4.2.1 Introduction
A fundamental characteristic of ANs is the ability to dynamically change their configuration and properties. In order to
describe the testing methodology some basic concepts (configuration states, state transitions and attractors) have to be
introduced, as their meaning is new or goes beyond well-known definitions for conventional networks.
4.2.2 Network States
A network is characterized by its hardware and software components, together with the configuration of these
components. This configuration is given by control elements, which can be on hardware level (e.g. elements
determining physical orientation of antennas) or on software level (parameters determining the functional behaviour of
a component). A component can have multiple control elements which define its overall state. Similarly, the overall
network state is defined by the overall states of each component. The total number of these controls - counting each
degree of freedom separately - is typically large, but finite and a fixed property of a given network.
Each degree of freedom can be:
• a discrete value, out of a given set of choices or a range of integer values; or
• a continuous (analogue) value.
The totality of all degrees of freedom represents the settings space. Each combination of settings can be described as an
N-dimensional vector, where N is the number of degrees of freedom, also called the dimension of the settings space. An
individual control setting is then the i-th element of this settings vector.
Each possible combination of settings is represented by the corresponding vector. For the purpose of the present
document, such a vector is termed Overall Configuration State (OCS).
A change of settings - regardless if done by human operators as in conventional networks or by automatic processes in
AN - means a transition between an initial OCS S to a new OCS S .
1 2
Resource A Configuration 1 Resource A Configuration 2
Control A_a: {a1} Control A_a: {a5}
C C
o o
n n
t t
r r
o o
l l
A A
Resource Resource
_ _
b b
A A
: :
{ {
b b
7 7
} }
Control A_c: {c3}
Control A_c: {c1}
Resource Resource Resource Resource Resource Resource
A B N A B N
{}. {.
OCS S OCS S
1 2
Figure 2: Concept of controls and Overall Configuration State (OCS) transitions
Also for this purpose and later usage, the term overall network properties (ONP) is defined which describes the
appearance of the network as perceivable from the end user point of view or through other interfaces to the network
operator (see also clause 5.1). Each OCS leads to a specific ONP.
NOTE: This relation is not symmetric; several OCS can lead to the same ONP, but the assumption is that the
same OCS cannot lead to different ONP. If this was the case it would mean that some aspect of the
network shows random behaviour which is a primarily unwanted condition.
ETSI
12 Final draft ETSI EG 203 341 V1.1.1 (2016-08)
4.2.3 Static and stationary states
In a conventional network, where controls are operated by humans, it is likely that settings, once made, do not
frequently change after they have been made. In contrast, in ANs, settings and associated network properties constantly
change as a result of various AF that operate in the network.
NOTE 1: In the context of the present clause, the term "state" represents the OCS, as introduced in clause 4.2.2.
S1
st atic
S2
stationary
Figure 3: Explanation of static and stationary states
To describe this situation, figure 3 shows a two-dimensional state space with two entities, S1 and S2.
S1 and S2 represent two types of states. A state which is constant over time (S1) is called static. A state which
fluctuates over time, around an identifiable point in the state space is called stationary (S2).Independent of the actual
shape or distribution of values, the essential property of a stationary state is that fluctuations occur within a given area,
which is sufficiently small compared to the overall state space.
NOTE 2: The definition of "small" is of course somewhat arbitrary. A pragmatic definition may be that effects on
the ONP are small against measurement errors in determining these properties.
The time scale of fluctuations is also an important characteristic of a stationary state. It will depend on both the
properties and capabilities of respective control elements and the characteristics of the decision processes in operation.
For later reference, state changes are called "microscopic" if they do not have a practical effect on ONP, and
"macroscopic" if they do.
In this state picture, instability either means large cyclical or chaotic fluctuations of the OCS with observable effects on
ONP, or a network state which is pulled towards some state with unwanted (unusable) ONP. Clearly, to determine the
temporal behaviour requires time which is - in addition to statistical reasons on sample number - the reason why such
measurements need appropriate time spans to perform.
4.2.4 State Transitions and Attractors
After having introduced the concept of static and stationary states, the question is how a NUT might change its state in
the course of the adaptation process. For illustration see figure 4.
ETSI
13 Final draft ETSI EG 203 341 V1.1.1 (2016-08)
S1
st atic
S2
stationary
Figure 4: Examples of state transition paths
Here, S1 is the state before the adaptation starts (initial state), and S2 is the state after the adaptive process is completed
(end state). The figure shows two paths from S1 to S2, a direct one and an indirect one. The actual path depends on the
adaptive algorithms being used. Even if it appears unlikely that a NUT actually shows a behaviour as the one shown in
this example, it cannot be excluded either. It may be the consequence of restrictions in network resource control or of
actual properties of the algorithms used. Also, it is conceivable that such a behaviour is, in distributed adaptive
networks, the result of interplay between "local" actions.
From the association between internal network configuration (OCS) and network performance (ONP), it follows that
during a state transition, the QoS of the network may be degraded. From the testing viewpoint, this has to be considered
too. While such a temporary degradation may be unavoidable in general, the impact as seen from a network subscriber's
perspective will depend on its duration and seriousness. Therefore, respective properties need to be considered in the
functional targets and assessment procedures used in testing. For example - in case of comparative testing or
benchmarking of two ANs, one candidate may exhibit a faster adaptation process, or an adaptation towards a better end
state while exhibiting a more serious or longer period of degradation than the other.
Basically, a network can have any state physically or technically possible, i.e. the initial state can be any point in state
space. If an adaptive process sets in, the state will - if the NUT is not unstable - move towards the end state. As the
adaptive process is actually an optimization of network parameters, there will be a finite number of end states, each of
which represents a local optimum, or the global optimum of the network with respect to the targets given by
administrative policies and current operating conditions. Figure 5 shows an example.
ETSI
14 Final draft ETSI EG 203 341 V1.1.1 (2016-08)

Figure 5: State space with local optima shown as end states
The points represent examples for initial states. If the NUT is brought into one of these states - and if such a state does
not already represent a stable or stationary state - and adaptation is enabled, the adaptation will lead, according to the
adaptation algorithm, to an end state. The concept of optimization implies that most of the possible states are not
optimal with respect to the rule set applied. Therefore, a transition from the initial state to a "better" state will occur.
Under the assumption that there are multiple (local) optima, the state space will have regions of initial states which lead
to different (optimized) end states.
NOTE: The state space may have characteristics which do not allow a direct path from a given starting point to
the global optimum of the system, by applying an incremental (mathematical, e.g. gradient based)
optimization algorithm.
Initial states in the left region lead to the end state E1; initial states in the right region lead to end state E2. In analogy to
the use of this term in other areas E1 and E2 are called attractors of the system.
The shape of the attractor landscape may have considerable effect on the dynamic behaviour and the predictability of
the system in the field. Assume a situation with a complex or rather fragmented attractor space. Two starting points
which are close to each other in the state space, may have attractors associated with quite different network
configurations. In a laboratory environment the degree of control over the starting points is higher compared to
operational networks. In effect, this may limit the ability to predict which configurations will be reached in actual
operation.
The situations described above are idealized by assuming that during the time an adaptive process is taking place, the
conditions which had caused this adaptation remain constant. If conditions change during this transition, and
considering a fragmented attractor space, there may be a high probability that the system is oscillating between end
states with probable negative effects on QoS. The test strategy should define means to detect such situations.
4.3 Adaptive Networks as Network Under Test
Testing ANs, implies testing a system of AFs that operate towards meeting functional targets defined by the network
policies. The scenario can be compared to traditional interoperability testing, where the goal is to verify the end-to-end
functionality (as experienced by a user) of several Functions Under Test (FUT).
Individual AFs typically pass through a conformance testing procedure at design time. However, AFs may be coupled
and interact during operation, leading to situations that were not anticipated beforehand. Consequently, testing
individual (or subsets) of AFs may not guarantee proper end-to-end functionality of the AN, unless it can be ensured
that the tested function (or subset) is independent from other AFs that operate in the same AN.
ETSI
15 Final draft ETSI EG 203 341 V1.1.1 (2016-08)
AF1 AF3
AF4
AF2
Figure 6: Example of NUT comprised of several AFs and potential interactions
The complexity of the NUT is given by the number of AFs it consists of, the hierarchical level at which they operate
and are aggregated and also the time scales at which they operate. AFs may be standalone or interconnected with other
AFs.
For testing purposes, it may be helpful to split a complex NUT into smaller segments. However, it is essential that any
split does not impact control loops, in order to avoid altering the dynamic behaviour of the NUT. Potential criteria that
may be used to segment an NUT are:
• Hierarchical aggregation level - the adaptive NUT will be tested only at a specific aggregation level.
• Time scale - the adaptive NUT will be tested only for adaptive functions that operate on a certain time scale,
e.g. slow control loops.
• Functional target - the adaptive NUT will be tested only towards a certain number of functional targets.
However, as discussed above, testing different segments of the NUT may not be equivalent to testing the NUT from an
end-to-end perspective.
5 Entities and interactions
5.1 Overview
Figure 7 shows the principal testing environment for adaptive network testing. It consists of:
• the network under test (NUT);
• effectors which constitute the stimuli for testing;
• sensors which provide information about the NUT;
• optional monitoring points for internal NUT information.
ETSI
16 Final draft ETSI EG 203 341 V1.1.1 (2016-08)
Monitoring Information
S
A1
R
O
T
C A2
E
F
F
E
NUT
S I1
R
O
S
I2
N
E
S
Figure 7: Testing environment
The test control domain is not shown in figure 7. The NUT is treated as a solid "black box" from a dynamic
(behavioural) point of view; see clause 4 for assumptions about its inner structure. With respect to the (logical)
architecture, assumption is that it can be described, on an abstract level, by a generic model, e.g. GANA. Given the
nature and current state of development of ANs, any further assumptions about architectural or structural details should
be avoided as these may be misleading.
On the effector side, activity type A1 denotes activities which are equivalent or identical to those coming, in real
network operation, from end users. They include all types of traffic that can be applied to the network, e.g. audio or
video calls (e.g. VoLTE or legacy telephony) as well as usage of data services such as Web Browsing, video streaming
and other types of packet data based activities. A1 activities may include also any form of machine-type traffic relevant
to the tested network.
Activity type A2 is the category for actions towards the NUT which cannot be triggered by end users. They include
structural actions such as addition or removal of physical or virtualized network elements. Also, A2 activities include
policy modifications or changes in defined functional targets that cause adaptive functions to change network settings or
behaviour.
Likewise, on the sensor side two general types of information are distinguished:
• Sensor information type I1 is information related to properties of the network which are visible to subscribers,
i.e. QoS information such as accessibility, retainability, throughpu
...