ETSI TS 102 250-6 V1.3.1 (2019-11)

TECHNICAL SPECIFICATION
Speech and multimedia Transmission Quality (STQ);
QoS aspects for popular services in mobile networks;
Part 6: Post processing and statistical methods

2 ETSI TS 102 250-6 V1.3.1 (2019-11)

Reference
RTS/STQ-00224-6m
Keywords
3G, GSM, network, QoS, service, speech
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: +33 4 92 94 42 00  Fax: +33 4 93 65 47 16

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
The present document can be downloaded from:
http://www.etsi.org/standards-search
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the prevailing version of an ETSI
deliverable is the one made publicly available in PDF format at www.etsi.org/deliver.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
https://portal.etsi.org/TB/ETSIDeliverableStatus.aspx
If you find errors in the present document, please send your comment to one of the following services:
https://portal.etsi.org/People/CommiteeSupportStaff.aspx
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© ETSI 2019.
All rights reserved.
DECT™, PLUGTESTS™, UMTS™ and the ETSI logo are trademarks of ETSI registered for the benefit of its Members.

3GPP™ and LTE™ are trademarks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
oneM2M™ logo is a trademark of ETSI registered for the benefit of its Members and
of the oneM2M Partners. ®
GSM and the GSM logo are trademarks registered and owned by the GSM Association.
ETSI
3 ETSI TS 102 250-6 V1.3.1 (2019-11)
Contents
Intellectual Property Rights . 6
Foreword . 6
Modal verbs terminology . 6
Introduction . 6
1 Scope . 7
2 References . 7
2.1 Normative references . 7
2.2 Informative references . 7
3 Definition of terms, symbols and abbreviations . 8
3.1 Terms . 8
3.2 Symbols . 8
3.3 Abbreviations . 8
4 Important measurement data types in mobile communications . 9
4.1 Data with binary values . 9
4.2 Data out of time-interval measurements . 9
4.3 Measurement of data throughput . 10
4.4 Data concerning quality measures . 10
5 Distributions and moments . 10
5.1 Introduction . 10
5.2 Continuous and discrete distributions. 11
5.3 Definition of density function and distribution function . 11
5.3.1 Probability Distribution Function (PDF) . 11
5.3.2 Cumulative Distribution Function (CDF) . 12
5.4 Moments and quantiles . 12
5.5 Estimation of moments and quantiles . 14
5.6 Important distributions . 15
5.6.1 Overview of distributions . 15
5.6.1.1 Continuous distributions . 15
5.6.1.2 Normal distribution . 16
5.6.1.3 Standard normal distribution . 16
5.6.1.4 Central limit theorem . 17
5.6.1.5 Transformation to normality . 17
5.6.1.6 Log-Normal distribution . 17
5.6.1.7 Use-case: transformations . 18
5.6.1.8 Exponential distribution . 18
5.6.1.9 Weibull distribution . 19
5.6.1.10 Pareto distribution . 20
5.6.1.11 Extreme distribution (Fisher-Tippett distribution) . 21
5.6.2 Overview of testing distributions . 21
5.6.2.1 Chi-Square distribution with n degrees of freedom . 21
5.6.2.2 Further relations . 23
5.6.2.3 Relation to empirical variance. 23
5.6.2.4 Student t-distribution . 23
5.6.2.5 Relation to normal distribution . 24
5.6.2.6 F distribution . 25
5.6.2.7 Quantiles . 26
5.6.2.8 Approximation of quantiles . 26
5.6.2.9 Relations to other distributions . 27
5.6.3 Overview of discrete distributions . 27
5.6.3.1 Bernoulli distribution . 27
5.6.3.2 Binomial distribution . 28
5.6.3.3 Geometric distribution . 29
5.6.3.4 Poisson distribution . 30
ETSI
4 ETSI TS 102 250-6 V1.3.1 (2019-11)
5.6.4 Overview of transitions between distributions and appropriate approximations . 31
5.6.4.1 From binomial to Poisson distribution . 31
5.6.4.2 From binomial to Normal distribution . 31
5.6.4.3 From Poisson to Normal distribution . 31
5.6.5 Truncated distributions . 32
5.6.6 Overview of distribution selection and parameter estimation . 32
5.6.6.1 Test procedures . 32
5.6.6.2 Chi-Square test . 32
5.6.6.3 Kolmogorov-Smirnov test . 32
5.6.6.4 Shapiro-Wilk test . 32
5.6.6.5 Parameter estimation methods . 33
5.7 Evaluation of measurement data . 33
5.7.1 Statistical tests . 33
5.7.1.1 Formulation of statistical tests. 33
5.7.1.2 Classes of statistical tests . 34
5.7.1.3 Tests for normal and binomial data . 34
5.7.1.4 One-sample tests for normal data . 34
5.7.1.5 Two-sample tests for normal data . 35
5.7.1.6 Test for binomial data . 36
5.7.1.7 Distribution-free tests for location . 37
5.7.1.8 Sign tests . 37
5.7.1.9 Sign rank test . 37
5.7.1.10 Wilcoxon rank sum test . 38
5.7.2 Confidence interval . 38
5.7.2.1 Binomial distribution . 38
5.7.2.2 Normal (Gaussian) distribution . 40
5.7.3 Required sample size for certain confidence levels . 41
6 Visualization techniques. 42
6.1 Visualization of static data . 42
6.1.1 Histograms . 42
6.1.2 Barplots . 42
6.1.3 QQ-Plots . 43
6.1.4 Boxplots . 43
6.2 Visualization of dynamic data . 44
6.2.1 Line Diagrams . 44
6.2.2 Temporal changing Boxplots . 44
6.2.3 MMQ-Plots . 45
7 Time series modelling . 45
7.1 Descriptive characterization . 45
7.1.1 Empirical moments . 45
7.1.2 Decomposition of time series. 48
7.1.3 Determination of the trend component . 49
7.1.3.1 Trend function types . 49
7.1.3.2 Linear trend function . 50
7.1.3.3 Polynomial trend function . 50
7.1.3.4 Non-linear trend models . 50
7.1.3.5 Trend estimation . 51
7.1.3.6 Transformation of time series by filtering . 52
7.1.3.7 Linear filters . 52
7.1.3.8 Exponential filters . 54
7.1.4 Seasonal component . 55
8 Data aggregation . 56
8.1 Basic data aggregation operators . 56
8.2 Data sources, structures and properties . 57
8.2.1 Raw, Performance and Event data . 57
8.2.2 Key Performance Indicators / Parameters . 58
8.3 Aggregation hierarchies . 58
8.3.1 Temporal aggregation . 58
8.3.2 Spatial aggregation . 59
8.4 Parameter estimation methods . 59
ETSI
5 ETSI TS 102 250-6 V1.3.1 (2019-11)
8.4.1 Projection method . 59
8.4.2 Substitution method . 59
8.4.3 Application of estimation methods . 60
8.4.4 Attributes of aggregation operators . 60
8.5 Weighted aggregation . 61
8.5.1 Perceived QoS . 61
8.5.2 Weighted quantiles . 62
8.6 Additional data aggregation operators . 63
8.6.1 MAWD and BH . 63
8.6.2 AVGn . 63
9 Assessment of performance indices . 63
9.1 Estimation of performance parameters based on active service probing systems . 63
9.2 Monitoring concepts . 63
9.2.1 Control charts . 63
9.2.2 Other alarming rules . 64
9.3 Methods for evaluation of objectives . 64
9.3.1 Desirability functions . 64
9.3.2 Loss functions . 65
Annex A (informative): Examples of statistical calculations . 66
A.1 Overview . 66
A.1.1 Step by step computation . 66
A.1.2 Computation using statistical software . 68
A.2 Transition from binomial to normal distribution . 69
A.3 Definitions of ETSI EG 201 769 [i.1] . 70
A.4 Calculation of confidence intervals . 70
A.4.1 Estimated rate 5 % . 70
A.4.2 Estimated rate 50 % . 71
A.4.3 Estimated rate 95 % . 72
A.4.4 Lower limit of confidence intervals according to Pearson-Clopper formula . 73
A.4.5 Upper limit of confidence intervals according to Pearson-Clopper formula . 74
A.4.6 Span of confidence intervals according to Pearson-Clopper formula . 75
A.5 Different sample sizes . 77
A.6 Calculation methods . 79
A.6.1 Calculation of quantiles . 79
A.7 Reporting of results . 80
A.7.1 Methods to use . 80
A.7.2 Number of significant decimals . 82
A.7.3 Rounding of end results . 82
Annex B (informative): Bibliography . 83
History . 84

ETSI
6 ETSI TS 102 250-6 V1.3.1 (2019-11)
Intellectual Property Rights
Essential patents
IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (https://ipr.etsi.org/).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Trademarks
The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners.
ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no
right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does
not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.
Foreword
This Technical Specification (TS) has been produced by ETSI Technical Committee Speech and multimedia
Transmission Quality (STQ).
The present document is part 6 of a multi-part deliverable. Full details of the entire series can be found in part 1 [i.2].
Modal verbs terminology
In the present document "shall", "shall not", "should", "should not", "may", "need not", "will", "will not", "can" and
"cannot" are to be interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of
provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
Introduction
All the defined quality of service parameters and their computations are based on field measurements. That indicates
that the measurements were made from users point of view (full end-to-end perspective, taking into account the needs
of testing).
It is assumed that the end user can handle his mobile and the services he wants to use (operability is not evaluated at this
time). For the purpose of measurement it is assumed:
• that the service is available and not barred for any reason;
• routing is defined correctly without errors; and
• the target subscriber equipment is ready to answer the call.
Speech quality values measured should only be employed by calls ended successfully for statistical analysis.
However, measured values from calls ended unsuccessfully (e.g. dropped) should be available for additional evaluations
and therefore, need to be stored.
Further preconditions may apply when reasonable.
ETSI
7 ETSI TS 102 250-6 V1.3.1 (2019-11)
1 Scope
The present document describes definitions and procedures to be used for statistical calculations which are related to
Quality of Service (QoS) measurements done by serving probing systems in mobile communications networks,
especially GSM and 3G networks. Network performance measurements and their related post-processing are only
marginally covered in the present document.
2 References
2.1 Normative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
Referenced documents which are not found to be publicly available in the expected location might be found at
https://docbox.etsi.org/Reference/.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are necessary for the application of the present document.
Not applicable.
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] ETSI EG 201 769: "Speech Processing, Transmission and Quality Aspects (STQ); QoS parameter
definitions and measurements; Parameters for voice telephony service required under the ONP
Voice Telephony Directive 98/10/EC".
[i.2] ETSI TS 102 250-1: "Speech and multimedia Transmission Quality (STQ); QoS aspects for
popular services in mobile networks; Part 1: Assessment of Quality of Service".
[i.3] NIST/SEMATECH: "e-Handbook of Statistical Methods".
NOTE: Available at http://www.itl.nist.gov/div898/handbook/, retrieved 17 September 2019.
[i.4] A. M. Law, W. D. Kelton: "Simulation modeling and analysis", McGraw-Hill, 3rd edition, 2000.
[i.5] J. Hartung: "Lehr- und Handbuch der angewandten Statistik", Oldenbourg Wissenschaftsverlag,
13th meditation, 2002.
[i.6] Bates, D.M. and Chambers, J.M: "Nonlinear regression Analysis and Applications", Wiley &
Sons, 1988.
[i.7] Mood, Graybill, Boes: "Introduction to the theory of statistics", McCraw-Hill Statistics Series,
1974.
ETSI
8 ETSI TS 102 250-6 V1.3.1 (2019-11)
[i.8] Venables, W.N. and Ripley, B.D.: "Modern Applied Statistics with S-Plus", Springer Verlag,
1999.
3 Definition of terms, symbols and abbreviations
3.1 Terms
For the purposes of the present document, the following terms apply:
rate: measurement result which is related to the portion of time during which it has been executed
NOTE: The denominator's unit is related to time.
ratio: measurement result which quantifies how a subgroup of all single measurements is related to the total number of
executed single measurements
NOTE: Usually, nominator and denominator share the same unit, namely a counter for measurements
(subgroup/all).
3.2 Symbols
For the purposes of the present document, the following symbols apply:
E(x)=µ Expected value of random variable x
Var(x)=σ Variance of random variable x
σ Standard deviation of random variable x
f(x) Probability Density Function (PDF) of random variable x
F(x) Cumulative Distribution Function (CDF) of random variable x
S, x∈ S Set of discrete values or interval of values the random variable x may take
IR Set of real numbers
2 2
s, s Empirical standard deviation / variance, analogous to σ and σ (theoretical)
q α-Quantile
α
u α-Quantile of standard normal distribution
α
x , x , x i-th ordered value, minimum and maximum of a given data set x , i =1,.,n
(i) (1) (n) i
3.3 Abbreviations
For the purposes of the present document, the following abbreviations apply:
3G Third Generation
ARMA Auto-Regressive Moving Average
AVGn Averaging Operator (regarding n days)
BH Busy Hour
BSC Base Station Controller
CDF Cumulative Distribution Function or Cumulative Density Function
CUSUM CUmulated SUM
EWMA Exponentially Weighted Moving Average
GSM Global System for Mobile communications
KPI Key Performance Indicator
LSL Lower Specification Level
MAWD Monthly Average Working Day
MMQ-Plot Median-Mean-Quantile Plot
MMS Multimedia Messaging Service
MOS Mean Opinion Score
MSC Mobile Switching Centre
NE Network Element
PDF Probablility Distribution Function or Probability Density Function
ETSI
9 ETSI TS 102 250-6 V1.3.1 (2019-11)
QoS Quality of Service
QQ-Plot Quantile-Quantile Plot
SMS Short Message Service
USL Upper Specification Level
4 Important measurement data types in mobile
communications
4.1 Data with binary values
Appropriate data analysis methods should depend on the type of the given data as well as on the scope of the analysis.
Therefore before analysis methods are described, different data types are introduced and differences between them are
pointed out.
Four general categories of measurement results are expected when QoS measurements are done in mobile
communications.
Single measurements related to the data with binary values:
• service accessibility, service availability;
• service retainability, service continuity;
• error ratios, error probabilities;
in general show a binary outcome, i.e. only two outcomes are possible. This means the result of a single trial leads to a
result which is either valued positive or negative related to the considered objective. The result may be recorded as
decision-results Yes / No or True / False or with numerical values 0 = successful and 1 = unsuccessful (i.e. errors occur)
or vice versa. Aggregation of trials of both types allows to calculate the according ratios which means the number of
positive / negative results is divided by the number of all trials. Usually, the units of nominator and denominator are the
same, namely number of trials.
EXAMPLE: If established speech calls are considered to test the service retainability of a speech telephony
system, every successfully completed call leads to the positive result "Call completed", every
unsuccessfully ended call is noticed as "Dropped call" which represents the negative outcome.
After 10 000 established calls, the ratio of dropped calls related to all established calls can be
calculated. The result is the call drop probability.
4.2 Data out of time-interval measurements
Measurements related to the time domain occur in the areas:
• duration of a session or call;
• service access delay;
• round trip time and end-to-end delay of a service;
• blocking times, downtimes of a system.
The outcome of such measurements is the time span between two time stamps marking the starting and end point of the
time periods of interest. Results are related to the unit "second" or multiples or parts of it. Depending on the
measurement tools and the precision needed, arbitrarily small measurement units may be realized.
EXAMPLE: Someone can define the end-to-end delivery time for the MMS service by a measurement which
starts when the user at the A party pushes the "Send" button and which stops when the completely
received MMS is signalled to the user at the B party.
ETSI
10 ETSI TS 102 250-6 V1.3.1 (2019-11)
4.3 Measurement of data throughput
Measurements related to data throughput result in values which describe the ratio of transmitted data volume related to
the required portion of time. The outcome of a single measurement is the quotient of both measures. Used units are "bit"
or multiples thereof for the data amount and "second" or multiples or parts thereof for the portion of time.
EXAMPLE: If a data amount of 1 Mbit is transmitted within a period of 60 seconds, this results in a mean data
rate of approximately 16,66 kbit/s.
4.4 Data concerning quality measures
Examples are given by the quality of data transfer which may be measured by its speed or evaluations of speech quality
measured on a scale, respectively.
Measurements related to audio-visual quality can be done objectively by algorithms or subjectively by human listeners.
The outcome of audio-visual quality evaluation is related to a scaled value which is called Mean Opinion Score (MOS)
for subjective testing. Thereby two types of quality measurement are distinguished subjective and objective
measurements. If quantitative measures are identified which are highly correlated to the quality of interest, this will
simplify the analysis. However, if this is not possible, some kind of evaluation on a standardized scale by qualified
experts is needed. The result may therefore be given either as the measurement result or as a mark on a pre-defined
scale.
EXAMPLE: Within a subjective test, people are asked to rate the overall quality of video samples which are
presented to them. The allowed scale to rate the quality is defined in the range from 1 (very poor
quality) to 5 (brilliant quality).
Table 4.1 summarizes the different kinds of QoS related measurements, typical outcomes and some examples.
Table 4.1: QoS related measurements, typical outcomes and examples
Category Relevant measurement types Examples
Binary values Service accessibility, service availability Service accessibility telephony, service
non-availability SMS
Service retainability, service continuity Call completion rate, call drop rate
Error ratios, error probabilities Call set-up error rate
Duration values Duration of a session or call Mean call duration
Service access delay Service access delay WAP
Round trip time, end-to-end delay ICMP Ping roundtrip time
Blocking times, system downtimes Blocking time telephony, SGSN downtime
Throughput values Throughput Mean data rate GPRS
Peak data rate UMTS
Content quality values Audio-visual quality MOS scores out of subjective testing

5 Distributions and moments
5.1 Introduction
The objective of data analyses is to draw conclusions about the state of a process based on a given data set, which may
or may not be a sample of the population of interest. If distributions are assumed, these specify the shape of the data
mass up to parameters associated with each family of distributions specifying properties like the mean of the data mass.
Location or dispersion shifts of the process will in general result in different parameter estimates specifying the
distribution. Therefore the information available from the data is compressed into one or few sufficient statistics
specifying the underlying distribution.
Many statistical applications and computations rely in some sense on distributional assumptions, which are not always
explicitly stated. Results of statistical measures are often only sensible if underlying assumptions are met and therefore
only interpretable if users know about these assumptions.
ETSI
11 ETSI TS 102 250-6 V1.3.1 (2019-11)
This clause is organized as follows. Firstly, distributions, moments and quantiles are introduced in theory in
clauses 5.2 to 5.4. This part of the present document is based on the idea of random variables having certain
distributions. Random variables do not take single values but describe the underlying probability model of a random
process. They are commonly denoted by:
X ~ distribution (parameters)
From the distributional assumptions, moments and quantiles of random variables are derived in theory.
Data is often viewed as being realizations of random variables. Therefore, data analysis mainly consists of fitting an
appropriate distribution to the data and drawing conclusions based on this assumption. Clause 5.5 briefly summarizes
the estimation of moments and quantiles.
Subsequently, a number of important distributions is introduced in clause 5.6, each of which is visualized graphically to
give an idea of meaningful applications. Within this clause, testing distributions are also introduced as they are needed
in clause 5.7 for the derivation of statistical tests.
5.2 Continuous and discrete distributions
The main difference between the data types described above can be explained in terms of continuous and discrete
distributions. Data with binary values follow a discrete distribution, since the probability mass is distributed only over a
fixed number of possible values. The same holds for quality measurements with evaluation results on a scale with a
limited number of possible values (i.e. marks 1 to 6 or similar).
On the contrary, time-interval measurements as well as quality measurements based on appropriate quantitative
variables may take an infinitely large number of possible values. In theory, since the number of possible outcomes
equals infinity, the probability that a single value is exactly realized is zero. Probabilities greater than zero are only
realized for intervals with positive width. In practice, each measurement tool will only allow a limited precision
resulting in discrete measurements with a large number of possible outcomes. Nevertheless, data from measurement
systems with reasonable precision are treated as being continuous.
Formal definitions for continuous and discrete distributions are based on probability density functions as described in
the following clauses.
5.3 Definition of density function and distribution function
5.3.1 Probability Distribution Function (PDF)
Probability Density Functions (PDF) specify the probability mass either for single outcomes (discrete distributions) or
for intervals (continuous distributions).
A PDF is defined as a function f : IR → [0, ∞) with properties:
i) f (x) ≥ 0 for all x∈ S.
ii) for continuous distributions or f (x) =1 for discrete distributions.
f (x)dx = 1

 S
S
In other words, firstly the values of the PDF are always non-negative, meaning that negative probabilities are neither
assigned to values nor intervals, and secondly the summation or integration over the PDF always results in 1 (= 100 %),
meaning that any data value will always be realized.
 0,1 : x = 1
EXAMPLE 1: A PDF for binary data may be given by , which implies that the probability for
f (x) =

0,9 : x = 0

a faulty trial (x=1) is 10 %, while tests are completed successfully with probability 90 %.
EXAMPLE 2: For time-interval measurements PDFs may take a
...