ETSI TS 103 281 V1.3.1 (2019-05)

ETSI TS 103 281 V1.3.1 (2019-05)






TECHNICAL SPECIFICATION
Speech and multimedia Transmission Quality (STQ);
Speech quality in the presence of background noise:
Objective test methods for super-wideband and
fullband terminals



---------------------- Page: 1 ----------------------
2 ETSI TS 103 281 V1.3.1 (2019-05)



Reference
RTS/STQ-269
Keywords
noise, quality, speech, testing, transmission

ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: +33 4 92 94 42 00  Fax: +33 4 93 65 47 16

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
The present document can be downloaded from:
http://www.etsi.org/standards-search
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the prevailing version of an ETSI
deliverable is the one made publicly available in PDF format at www.etsi.org/deliver.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
https://portal.etsi.org/TB/ETSIDeliverableStatus.aspx
If you find errors in the present document, please send your comment to one of the following services:
https://portal.etsi.org/People/CommiteeSupportStaff.aspx
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© ETSI 2019.
All rights reserved.

TM TM TM
DECT , PLUGTESTS , UMTS and the ETSI logo are trademarks of ETSI registered for the benefit of its Members.
TM TM
3GPP and LTE are trademarks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
oneM2M™ logo is a trademark of ETSI registered for the benefit of its Members and
of the oneM2M Partners.
®
GSM and the GSM logo are trademarks registered and owned by the GSM Association.
ETSI

---------------------- Page: 2 ----------------------
3 ETSI TS 103 281 V1.3.1 (2019-05)
Contents
Intellectual Property Rights . 6
Foreword . 6
Modal verbs terminology . 6
1 Scope . 7
2 References . 7
2.1 Normative references . 7
2.2 Informative references . 8
3 Definition of terms, symbols and abbreviations . 10
3.1 Terms . 10
3.2 Symbols . 10
3.3 Abbreviations . 10
4 Introduction . 11
5 Underlying speech databases and preparations . 11
6 Model descriptions . 12
6.1 Introduction . 12
6.2 Common definitions . 12
6.3 Model A . 12
6.3.1 Introduction. 12
6.3.2 Pre-Processing . 13
6.3.3 Spectral transformation . 14
6.3.4 Non-linear loudness transformation . 16
6.3.5 Instrumental assessment of N-MOS . 17
6.3.5.1 Introduction . 17
6.3.5.2 Loudness-based features . 17
6.3.5.3 Sharpness-based feature . 17
6.3.6 Reference optimization and asymmetry . 18
6.3.6.1 Introduction . 18
6.3.6.2 Reference optimization . 19
6.3.6.3 Masking of inaudible differences . 19
6.3.6.4 Asymmetry . 19
6.3.7 Instrumental assessment of S-MOS . 20
6.3.7.1 Introduction . 20
6.3.7.2 Modulation-based features . 20
6.3.7.3 Spectral difference features . 20
6.3.7.4 Control parameters . 21
6.3.7.5 Combination of features . 22
6.3.8 Instrumental assessment of G-MOS . 22
6.4 Model B . 23
6.4.1 Overview . 23
6.4.2 Operational Modes . 24
6.4.3 Temporal Alignment . 24
6.4.4 Voice Activity Detection (VAD) and segment classification . 25
6.4.5 Auditory Model . 25
6.4.5.1 Introduction . 25
6.4.5.2 Ear Canal model . 26
6.4.5.3 Middle Ear model . 26
6.4.5.4 Hydro-mechanical cochlear model . 27
6.4.5.5 Hair Cell transduction model . 27
6.4.5.6 Outer Hair motility model . 28
6.4.6 Feature Extraction . 28
6.4.6.1 Introduction . 28
ETSI

---------------------- Page: 3 ----------------------
4 ETSI TS 103 281 V1.3.1 (2019-05)
6.4.6.2 Salient Formant Points (SFP) feature extraction . 28
6.4.6.3 COSM (Cochlear Output Statistic Metric) feature extraction . 30
6.4.7 Training and mapping . 34
6.5 Mapping of model outputs . 35
7 Comparison of objective and subjective results after the training process . 36
7.1 Introduction . 36
7.2 Results for Model A . 36
7.3 Results for Cochlear Prediction Model (Model B) . 41
8 Validation results . 45
8.1 Introduction . 45
8.2 Validation database 1 (DES-17) . 45
8.2.1 Database description . 45
8.2.2 Validation database 1: Results for model B . 46
8.3 Validation database 2 (DES-20) . 48
8.3.1 Database description . 48
8.3.2 Validation database 2: Results for model A . 48
8.4 Validation database 3 (DES-25) . 50
8.4.1 Database description . 50
8.4.2 Validation database 3: Results for model A . 51
8.4.3 Validation database 3: Results for model B . 52
8.5 Validation database 4 (DES-26) . 54
8.5.1 Database description . 54
8.5.2 Validation database 4: Results for model A . 54
8.5.3 Validation database 4: Results for model B . 56
8.6 Validation database 5 (DES-27) . 58
8.6.1 Database description . 58
8.6.2 Validation database 5: Results for model A . 59
8.6.3 Validation database 5: Results for model B . 61
9 Application of the models . 63
9.1 Introduction . 63
9.2 Speech material . 63
9.3 Positioning of the device under test . 63
9.4 Background noise playback . 64
9.5 Recording and calibration procedure . 64
9.6 Running the prediction models . 64
9.7 Mapping function for Model A . 64
9.7.1 Derivation of mapping functions . 64
9.7.2 Resulting mapping functions . 68
Annex A (normative): Model configuration files . 69
A.1 Introduction . 69
A.2 Model A . 69
A.3 Model B . 70
Annex B (normative): Summary of Training Databases . 71
Annex C (normative): Test vectors for model verification . 73
Annex D (informative): Subjective testing framework . 74
D.1 Introduction . 74
D.2 Subjective test plan . 74
D.2.1 Traceability. 74
D.2.2 Speech database requirements . 74
D.2.3 Reference Conditions . 74
D.2.4 Test Conditions . 74
D.2.5 Post-processing of test conditions . 75
D.2.6 Calibration and equalization of headphones for presentation . 76
ETSI

---------------------- Page: 4 ----------------------
5 ETSI TS 103 281 V1.3.1 (2019-05)
D.2.7 Requirements on the listening laboratory . 76
D.2.8 Experimental design . 77
D.2.9 Training session . 77
D.3 Set-up for acquisition of test conditions . 77
D.3.1 Terminal positioning and HATS calibration . 77
D.3.2 Background Noise reproduction . 78
D.3.3 Noise and speech playback synchronization . 78
D.3.4 Convergence sequence . 78
D.3.5 Example of noise and speech playback sequence including convergence period . 78
D.3.6 Recordings at the network simulator electrical reference point . 79
D.3.7 Recordings at the MRP and terminal's primary microphone location . 79
Annex E (normative): Speech material to be used for objective testing . 80
History . 82

ETSI

---------------------- Page: 5 ----------------------
6 ETSI TS 103 281 V1.3.1 (2019-05)
Intellectual Property Rights
Essential patents
IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (https://ipr.etsi.org/).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Trademarks
The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners.
ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no
right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does
not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.
Foreword
This Technical Specification (TS) has been produced by ETSI Technical Committee Speech and multimedia
Transmission Quality (STQ).
The present document is to be used in conjunction with:
• ETSI ES 202 396-1 [i.1]: "Background noise simulation technique and background noise database"; and
• ETSI TS 103 224 [i.19] series: "A sound field reproduction method for terminal testing including a
background noise database".
The present document describes an objective test method for super-wideband and fullband in order to provide a good
prediction of the uplink speech quality in the presence of background noise of modern mobile terminals in hand-held
and hands-free.
Modal verbs terminology
In the present document "shall", "shall not", "should", "should not", "may", "need not", "will", "will not", "can" and
"cannot" are to be interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of
provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
ETSI

---------------------- Page: 6 ----------------------
7 ETSI TS 103 281 V1.3.1 (2019-05)
1 Scope
The present document describes testing methodologies which can be used to objectively evaluate the performance of
super-wideband and fullband mobile terminals for speech communication in the presence of background noise.
Background noise is a problem in mostly all situations and conditions and needs to be taken into account in terminal
design. The present document provides information about the testing methods applicable to objectively evaluate the
speech quality of mobile terminals (including any state-of-the-art codecs) employing background noise suppression in
the presence of background noise. The present document includes:
• The method which is applicable to objectively determine the different parameters influencing the speech
quality in the presence of background noise taking into account:
- the speech quality;
- the background noise transmission quality;
- the overall quality.
• The model results in comparison with the underlying subjective tests used for the training of the objective
model. The underlying languages are: American English, German, Chinese (Mandarin).
• The model validation results.
The present document is to be used in conjunction with:
• ETSI ES 202 396-1 [i.1] which describes a recording and reproduction setup for realistic simulation of
background noise scenarios in lab-type environments for the performance evaluation of terminals and
communication systems.
• ETSI TS 103 224 [i.19] which describes a sound field reproduction method for terminal testing including a
background noise database with background noise scenarios to be used in lab-type environments for the
performance evaluation of terminals and communication systems.
• American English speech sentences as enclosed in the present document.
2 References
2.1 Normative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
Referenced documents which are not found to be publicly available in the expected location might be found at
https://docbox.etsi.org/Reference/.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are necessary for the application of the present document.
Not applicable.
ETSI

---------------------- Page: 7 ----------------------
8 ETSI TS 103 281 V1.3.1 (2019-05)
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] ETSI ES 202 396-1: "Speech and multimedia Transmission Quality (STQ); Speech quality
performance in the presence of background noise; Part 1: Background noise simulation technique
and background noise database".
[i.2] ETSI EG 202 396-3: "Speech and multimedia Transmission Quality (STQ); Speech Quality
performance in the presence of background noise Part 3: Background noise transmission -
Objective test methods".
[i.3] ETSI TS 103 106: "Speech and multimedia Transmission Quality (STQ); Speech quality
performance in the presence of background noise: Background noise transmission for mobile
terminals-objective test methods".
[i.4] ETSI TS 126 441: "Universal Mobile Telecommunications System (UMTS); LTE; Codec for
Enhanced Voice Services (EVS); General overview (3GPP TS 26.441)".
[i.5] Recommendation ITU-T P.835: "Subjective test methodology for evaluating speech
communication systems that include noise suppression algorithm".
[i.6] Internet Engineering Task Force, Request for Comments 6716: "Definition of the Opus Audio
Codec", 09/2012.
[i.7] Recommendation ITU-T P.56: "Objective measurement of active speech level".
[i.8] Recommendation ITU-T P.1401: "Methods, metrics and procedures for statistical evaluation,
qualifying and comparison of objective quality prediction models".
[i.9] Recommendation ITU-T G.160 Appendix II, Amendment 2: "Voice enhancement devices:
Revised Appendix II - Objective measures for the characterization of the basic functioning of
noise reduction algorithms".
[i.10] Recommendation ITU-T P.501: "Test Signals for Use in Telephonometry".
[i.11] Recommendation ITU-T P.58: "Head and Torso simulator for telephonometry".
[i.12] Recommendation ITU-T P.57: "Artificial ears".
[i.13] Recommendation ITU-T P.800: "Methods for subjective determination of transmission quality".
[i.14] ETSI TS 126 132: "Universal Mobile Telecommunications System (UMTS); LTE; Speech and
video telephony terminal acoustic test specification (3GPP TS 26.132)".
[i.15] Recommendation ITU-T TD 477 (GEN/12): "Handbook of subjective test practical procedures"
(temporary document) - Geneva, 18-27 January 2011.
[i.16] AH-11-029: "Better Reference System for the P.835 SIG Rating Scale", Q7/12 Rapporteur's
meeting, 20-21 June 2011, Geneva, Switzerland.
[i.17] 3GPP, Tdoc S4(16)0397: "DESUDAPS-1: Common subjective testing framework for training and
validation of SWB and FB P.835 test predictors".
[i.18] Recommendation ITU-T P.64: "Determination of sensitivity/frequency characteristics of local
telephone systems".
ETSI

---------------------- Page: 8 ----------------------
9 ETSI TS 103 281 V1.3.1 (2019-05)
[i.19] ETSI TS 103 224: "Speech and multimedia Transmission Quality (STQ); A sound field
reproduction method for terminal testing including a background noise database".
[i.20] Sottek R.: "Modelle zur Signalverarbeitung im menschlichen Gehör", PHD thesis RWTH Aachen,
1993.
[i.21] Sot
...