ETSI TR 104 032 V1.1.1 (2024-02)

TECHNICAL REPORT
Securing Artificial Intelligence (SAI);
Traceability of AI Models
2 ETSI TR 104 032 V1.1.1 (2024-02)

Reference
DTR/SAI-004
Keywords
artificial intelligence, cyber security, digital right
management, ML watermarking, trustworthy AI

ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: +33 4 92 94 42 00  Fax: +33 4 93 65 47 16

Siret N° 348 623 562 00017 - APE 7112B
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° w061004871

Important notice
The present document can be downloaded from:
https://www.etsi.org/standards-search
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the prevailing version of an ETSI
deliverable is the one made publicly available in PDF format at www.etsi.org/deliver.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
https://portal.etsi.org/TB/ETSIDeliverableStatus.aspx
If you find errors in the present document, please send your comment to one of the following services:
https://portal.etsi.org/People/CommiteeSupportStaff.aspx
If you find a security vulnerability in the present document, please report it through our
Coordinated Vulnerability Disclosure Program:
https://www.etsi.org/standards/coordinated-vulnerability-disclosure
Notice of disclaimer & limitation of liability
The information provided in the present deliverable is directed solely to professionals who have the appropriate degree of
experience to understand and interpret its content in accordance with generally accepted engineering or
other professional standard and applicable regulations.
No recommendation as to products and services or vendors is made or should be implied.
No representation or warranty is made that this deliverable is technically accurate or sufficient or conforms to any law
and/or governmental rule and/or regulation and further, no representation or warranty is made of merchantability or fitness
for any particular purpose or against infringement of intellectual property rights.
In no event shall ETSI be held liable for loss of profits or any other incidental or consequential damages.

Any software contained in this deliverable is provided "AS IS" with no warranties, express or implied, including but not
limited to, the warranties of merchantability, fitness for a particular purpose and non-infringement of intellectual property
rights and ETSI shall not be held liable in any event for any damages whatsoever (including, without limitation, damages
for loss of profits, business interruption, loss of information, or any other pecuniary loss) arising out of or related to the use
of or inability to use the software.
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying and
microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© ETSI 2024.
All rights reserved.
ETSI
3 ETSI TR 104 032 V1.1.1 (2024-02)
Contents
Intellectual Property Rights . 5
Foreword . 5
Modal verbs terminology . 5
1 Scope . 6
2 References . 6
2.1 Normative references . 6
2.2 Informative references . 6
3 Definition of terms, symbols and abbreviations . 10
3.1 Terms . 10
3.2 Symbols . 10
3.3 Abbreviations . 10
4 Introduction . 11
4.1 The importance of traceability . 11
4.2 Traceability for ownership rights protection and fight against AI misuse . 11
4.3 Traceability for trustworthy AI . 11
5 Traceability for ownership rights protection . 12
5.1 Classical Digital Right Management . 12
5.1.0 Introduction. 12
5.1.1 Classical DRM mechanisms explained . 12
5.1.2 Protection of the training set . 13
5.1.3 Protection of the training parameters . 13
5.1.4 Protection of the architecture . 13
5.1.5 Protection of the ML system . 14
5.1.6 Protection of the model against copying . 14
5.1.7 Comparison of classical DRM mechanisms in the ML context . 14
5.2 AI-specific prevention of model misuse . 15
5.2.1 Full-knowledge exposure strategies . 15
5.2.2 Zero-knowledge exposure strategies . 16
5.3 ML watermarking . 16
5.3.0 Introduction to ML watermarking . 16
5.3.1 Verification setting and type . 16
5.3.1.0 Introduction . 16
5.3.1.1 Full-knowledge watermarking . 17
5.3.1.2 Zero-knowledge watermarking . 17
5.3.2 Payload . 18
5.3.3 Relation to model . 18
5.3.3.1 Model independent . 18
5.3.3.2 Model dependent . 18
5.3.4 Requirements for efficient ML watermarking . 19
5.4 Detection of training data misuse . 20
5.5 Threats against AI-specific tracing mechanisms . 20
5.5.0 Introduction. 20
5.5.1 Watermark removal . 21
5.5.2 Ownership check evasion . 22
5.5.3 Ambiguity attacks . 22
6 Traceability for trustworthy AI . 22
6.1 Classical provenance concepts in the context of AI . 22
6.2 AI-specific traceability needs . 23
6.2.1 Traceability of data . 23
6.2.2 Traceability of the processing pipeline . 24
6.2.3 Traceability of outputs . 25
6.2.3.0 Introduction to outputs traceability . 25
6.2.3.1 Types of concept drift . 25
ETSI
4 ETSI TR 104 032 V1.1.1 (2024-02)
6.2.4 Model metadata and lifecycle management . 26
6.2.5 Reproducibility of the training process . 26
6.2.5.1 Definition of reproducibility . 26
6.2.5.2 Evaluation and measurement of reproducibility. 27
6 Conclusion . 28
History . 29

ETSI
5 ETSI TR 104 032 V1.1.1 (2024-02)
Intellectual Property Rights
Essential patents
IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The declarations
pertaining to these essential IPRs, if any, are publicly available for ETSI members and non-members, and can be
found in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to
ETSI in respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the
ETSI Web server (https://ipr.etsi.org/).
Pursuant to the ETSI Directives including the ETSI IPR Policy, no investigation regarding the essentiality of IPRs,
including IPR searches, has been carried out by ETSI. No guarantee can be given as to the existence of other IPRs not
referenced in ETSI SR 000 314 (or the updates on the ETSI Web server) which are, or may be, or may become,
essential to the present document.
Trademarks
The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners.
ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no
right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does
not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.
DECT™, PLUGTESTS™, UMTS™ and the ETSI logo are trademarks of ETSI registered for the benefit of its

Members. 3GPP™ and LTE™ are trademarks of ETSI registered for the benefit of its Members and of the 3GPP
Organizational Partners. oneM2M™ logo is a trademark of ETSI registered for the benefit of its Members and of the ®
oneM2M Partners. GSM and the GSM logo are trademarks registered and owned by the GSM Association.
Foreword
This Technical Report (TR) has been produced by ETSI Technical Committee Securing Artificial Intelligence (SAI).
Modal verbs terminology
In the present document "should", "should not", "may", "need not", "will", "will not", "can" and "cannot" are to be
interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
ETSI
6 ETSI TR 104 032 V1.1.1 (2024-02)
1 Scope
The present document describes the role of traceability in the challenge of Securing AI and explores issues related to
sharing and re-using models across tasks and industries. The scope includes threats, and their associated remediations
where applicable, to ownership rights of AI creators as well as to verification of models origin. Mitigations can be non-
AI-Specific (Digital Right Management applicable to AI) and AI-specific techniques (e.g. ML watermarking) from
prevention and detection phases. They can be both model-agnostic and model enhancement techniques. The present
document aligns terminology with existing ETSI ISG SAI documents and studies, and references/complements
previously studied attacks and remediations (ETSI GR SAI 004 [i.2] and ETSI GR SAI 005 [i.3]). It also gathers
industrial and academic feedback on traceability and ownership rights protection and model verification in the context
of AI.
2 References
2.1 Normative references
Normative references are not applicable in the present document.
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] OpenAI's GPT-3 Language Model: A Technical Overview.
[i.2] ETSI GR SAI 004: "Securing Artificial Intelligence (SAI); Problem Statement".
[i.3] ETSI GR SAI 005: "Securing Artificial Intelligence (SAI); Mitigation Strategy Report".
[i.4] Artificial Intelligence Market Insights, 2020-2023. ®
[i.5] BankInfo Security : "To Combat Rogue AI, Facebook Pitches 'Radioactive Data".
[i.6] McKinney S. M. et al. (2020): "International evaluation of an AI system for breast cancer
screening". Nature 577, pp. 89-94.
[i.7] Intellectual Property aspects of Machine Learning, NXP, 2022.
[i.8] Foss-Solbrekk K.: "Three routes to protecting AI systems and their algorithms under IP law: The
good, the bad and the ugly". Journal of Intellectual Property Law & Practice 16.3 (2021),
pp. 247-258.
[i.9] Hu X., Liang L., Li S., Deng L., Zuo P., Ji Y., Xie X., Ding Y., Liu C., Sherwood T. & Xie Y.
2020: "DeepSniffer: A DNN Model Extraction Framework Based on Learning Architectural
Hints". In Proceedings of the Twenty-Fifth International Conference on Architectural Support for
Programming Languages and Operating Systems (ASPLOS '20). Association for Computing
Machinery, New York, NY, USA, pp. 385-399.
[i.10] Xu Q., Arafin Md T. & Qu G. (2021): "Security of Neural Networks from Hardware Perspective:
th
A Survey and Beyond". In Proceedings of the 26 Asia and South Pacific Design Automation
Conference (ASPDAC '21). Association for Computing Machinery, New York, NY, USA,
pp. 449-454.
ETSI
7 ETSI TR 104 032 V1.1.1 (2024-02)
[i.11] Chakraborty A., Mondal A. & Srivastava A. 2020: "Hardware-assisted intellectual property
th
protection of deep learning models". In Proceedings of the 57 ACM/EDAC/IEEE Design
TM
Press, Article 172, pp. 1-6.
Automation Conference (DAC '20). IEEE
[i.12] Goldstein B.F., Patil V.C., Ferreira V.C., Nery A.S., França F.M.G. & Kundu S.: "Preventing
TM
DNN Model IP Theft via Hardware Obfuscation". In IEEE Journal on Emerging and Selected
Topics in Circuits and Systems, vol. 11, no. 2, pp. 267-277, June 2021.
[i.13] Li J., He Z., Rakin A., Fan D. & Chakrabarti C. (2021): "NeurObfuscator: A Full-stack
Obfuscation Tool to Mitigate Neural Architecture Stealing", pp. 248-258.
10.1109/HOST49136.2021.9702279.
[i.14] Mahya Morid A., Alrahis L., Colucci A., Sinanoglu O. & Shafique M. (2022): "NeuroUnlock:
Unlocking the Architecture of Obfuscated Deep Neural Networks".
[i.15] Tramer F., Zhang F. J., Reiter M. & Ristenpart T. (2016): "Stealing machine learning models via
th
prediction apis. In 25 USENIX Security Symposium (USENIX Security 16), pp. 601-618.
[i.16] Kálmán S., Al-Afandi, J. & Horváth A. (2019): "MimosaNet: An Unrobust Neural Network
Preventing Model Stealing".
[i.17] Uchida Y., Nagai Y., Sakazawa S. & Satoh S. I. (2017): "Embedding watermarks into deep neural
networks". In Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval,
pp. 269-277.
[i.18] Adi Y., Baum C., Cisse M., Pinkas B. & Keshet J. (2018): "Turning your weakness into a strength:
th
Watermarking deep neural networks by backdooring". In 27 USENIX Security Symposium
(USENIX Security 18), pp. 1615-1631.
[i.19] Zhang J., Gu Z., Jang J., Wu H., Stoecklin M. P., Huang H. & Molloy I. (2018): "Protecting
intellectual property of deep neural networks with watermarking". In Proceedings of the 2018 on
Asia Conference on Computer and Communications Security, pp. 159-172.
[i.20] Fan L., Ng K. W. & Chan C. S. (2019): "Rethinking deep neural network ownership verification:
Embedding passports to defeat ambiguity attack. Advances in Neural Information Processing
Systems (NIPS)".
[i.21] Chen H., Rouhani B. D., Fu C. Z. & Koushanfar F. (2019): "Deepmarks: A secure fingerprinting
framework for digital rights management of deep learning models". In Proceedings of the 2019 on
International Conference on Multimedia Retrieval, pp. 105-113.
[i.22] Wang T. and Florian K. (2021) RIGA: "Covert and Robust White-Box Watermarking of Deep
Neural Networks". Proceedings of the Web Conference 2021.
[i.23] Li Z., Hu C., Zhang Y. & Guo S. (2019): "How to prove your model belongs to you: a blind-
watermark based framework to protect intellectual property of DNN". In Proceedings of the 35th
Annual Computer Security Applications Conference, pp. 126-137.
[i.24] Guo J. & Potkonjak M. (2018): "Watermarking deep neural networks for embedded systems". In
2018 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), pp. 1-8.
[i.25] Szyller S., Atli B., Marchal S. & Asokan N. (2019): "DAWN: Dynamic Adversarial Watermarking
of Neural Networks".
[i.26] Jia H., Choquette-Choo C. A. & Papernot N. (2020): "Entangled Watermarks as a Defense against
Model Extraction".
[i.27] Le Merrer E., Perez P. & Tredan G. (2019): "Adversarial frontier stitching for remote neural
network watermarking. Neural Computing and Applications", pp. 1-12.
[i.28] Lukas Nils, Yuxuan Zhang and Florian Kerschbaum. (2019): "Deep neural network fingerprinting
by conferrable adversarial examples".
[i.29] Cao Xiaoyu, Jinyuan Jia and Neil Zhenqiang Gong. "IPGuard: Protecting intellectual property of
deep neural networks via fingerprinting the classification boundary". Proceedings of the 2021
ACM Asia Conference on Computer and Communications Security.
ETSI
8 ETSI TR 104 032 V1.1.1 (2024-02)
[i.30] Namba R., & Sakuma J. (2019): "Robust watermarking of neural network with exponential
weighting". In Proceedings of the 2019 ACM Asia Conference on Computer and Communications
Security, pp. 228-240.
[i.31] Chen Huili, Bita Darvish Rouhani and Farinaz Koushanfar (2019): "BlackMarks: Blackbox
Multibit Watermarking for Deep Neural Networks".
[i.32] Sablayrolles A., Douze M., Schmid C. & Jegou H.: "Radioactive data: tracing through training". In
th
Proceedings of the 37 International Conference on Machine Learning, PMLR 119:8326-8335,
2020.
[i.33] Yang Z., Dang H. & Chang E. C. (2019): "Effectiveness of Distillation Attack and
Countermeasure on Neural Network Watermarking".
[i.34] Pan S. J. & Qiang Y. (2009): "A survey on transfer learning". IEEE Transactions on knowledge
and data engineering, pp. 1345-1359.
[i.35] Hinton G., Oriol V. & Jeff D. (2015): "Distilling the knowledge in a neural network. NIPS Deep
Learning and Representation Learning".
[i.36] Papernot N., Song S., Mironov I., Raghunathan A., Talwar K. & Erlingsson Ú. (2018): "Scalable
private learning with pate". arXiv:1802.08908.
[i.37] Chen X., Wang W., Bender C., Ding Y., Jia R., Li B. & Song D. (2019): "REFIT: a Unified
Watermark Removal Framework for Deep Learning Systems with Limited Data".
[i.38] Chen T., Goodfellow I. & Shlens J. (2016). Net2net: "Accelerating learning via knowledge
transfer". In Proceedings of ICLR.
[i.39] Wei T., Wang C., Rui Y. & Chen C. W. (2016): "Network morphism. In International Conference
on Machine Learning", pp. 564-572.
[i.40] Wang B., Yao Y., Shan S., Li H., Viswanath B., Zheng H., & Zhao B. (2019): "Neural Cleanse:
Identifying and Mitigating Backdoor Attacks in Neural Networks". pp.707-723.
10.1109/SP.2019.00031.
[i.41] Aiken W., Kim H. & Woo S.S. (2020): "Neural Network Laundering: Removing Black-Box
Backdoor Watermarks from Deep Neural Networks".
[i.42] Fredrikson M., Jha S. & Ristenpart T. (2015): "Model inversion attacks that exploit confidence
information and basic countermeasures". In Proceedings of the 22nd ACM SIGSAC Conference
on Computer and Communications Security, pp. 1322-1333.
[i.43] Kapusta K., Thouvenot V., Bettan O., Beguinet H. & Senet H. (2021): "A Protocol for Secure
Verification of Watermarks Embedded into Machine Learning Models". In Proceedings of the
2021 ACM Workshop on Information Hiding and Multimedia Security (IH&MMSec '21).
TM
[i.44] Kallas K. & Furon T. (2022): "ROSE: A RObust and SEcure DNN Watermarking". 2022 IEEE
International Workshop on Information Forensics and Security (WIFS), Shanghai, China, 2022,
pp. 1-6.
[i.45] Moreau L., Clifford B., Freire J., Futrelle J., Gil Y., Groth P., Kwasnikowska N., Miles S., Missier
P., Myers J., Plale B., Simmhan Y., Stephan E. & Van den Bussche J. (2010): "The open
provenance model core specification (v1.1)". Future Generation Computer Systems, July 2010.
[i.46] W3C Working Group Note 30 April 2013: "PROV-Overview - An Overview of the PROV Family
of Documents".
[i.47] ETSI GR SAI 002: "Securing Artificial Intelligence (SAI); Data Supply Chain Security".
[i.48] Gama J., Žliobaitė I., Bifet A. et al.: "A survey on concept drift adaptation. ACM computing
surveys (CSUR)", 2014, 46(4): pp. 1-37.
[i.49] Krawczyk B. & Cano A.: "Online ensemble learning with abstaining classifiers for drifting and
noisy data streams. Applied Soft Computing", 2018, 68: pp. 677-692.
ETSI
9 ETSI TR 104 032 V1.1.1 (2024-02)
[i.50] Kuncheva L. I.: "Change detection in streaming multivariate data using likelihood detectors".
IEEE transactions on knowledge and data engineering, 2011, 25(5): pp. 1175-1180.
[i.51] Bifet A. & Gavalda R. (2007): "Learning from time-changing data with adaptive windowing?"
Proceedings of the 2007 SIAM international conference on data mining. Society for Industrial and
Applied Mathematics, 2007: pp. 443-448.
[i.52] Zhang S., Pan C., Song L. et al. (2021): "Label-Assisted Memory Autoencoder for Unsupervised
Out-of-Distribution Detection. Joint European Conference on Machine Learning and Knowledge
Discovery in Databases". Springer, Cham, 2021: pp. 795-810.
[i.53] Wu, X., Hu, Z., Pei K, et al. (2021): "Methods for deep learning model failure detection and model
TM
adaption: A survey". IEEE International Symposium on Software Reliability Engineering
TM
Workshops (ISSREW). IEEE , 2021: 218-223.
[i.54] Mikołajczyk A. & Grochowski M. (2018): "Data augmentation for improving deep learning in
image classification problem. International interdisciplinary PhD workshop (IIPhDW)". IEEE,
pp. 117-122, 2018.
[i.55] Huang C., Hu Z., Huang X et al. (2021): "Statistical certification of acceptable robustness for
neural networks". International Conference on Artificial Neural Networks. Springer, Cham, 2021:
pp. 79-90.
[i.56] Shen Z., Cui P., Zhang T. et al. (2020): "Stable learning via sample reweighting". Proceedings of
the AAAI Conference on Artificial Intelligence, vol 34(04), pp. 5692-5699.
[i.57] Jo J., Verma V., Bengio Y. (2018): "Modularity matters: Learning invariant relational reasoning
tasks".
[i.58] Hummer W., Muthusamy V., Rausch T., Dube P., El Maghraoui K., Murthi A. & Oum P. (2019):
TM
"ModelOps: Cloud-Based Lifecycle Management for Reliable and Trusted AI". IEEE
International Conference on Cloud Engineering (IC2E), pp. 113-120.
[i.59] Vartak M., Subramanyam H., Lee W-E, Viswanathan S., Husnoo S., Madden S. & Zaharia M.
(2016): "MODELDB: A System for Machine Learning Model Management. Workshop on
Human-In-the-Loop Data Analytics (HILDA)", June 26 2016, San Francisco, CA, USA.
[i.60] Gundersen O. E. (2021): "The fundamental principles of reproducibility. Philosophical
Transactions of the Royal Society A", 379(2197), 20200210.
[i.61] Pineau J., Vincent-Lamarre P., Sinha K., Larivière V., Beygelzimer A., d'Alché-Buc F., Fox E.
and Larochelle H., 2021: "Improving reproducibility in machine learning research: a report from
the NeurIPS 2019 reproducibility program". Journal of Machine Learning Research, 22.
[i.62] Chen B., Wen M., Shi Y., Lin D., Rajbahadur G. K. & Jiang Z. M. (2022, May): "Towards
th
training reproducible deep learning models". In Proceedings of the 44 International Conference
on Software Engineering (pp. 2202-2214).
[i.63] Arcuri A. and Briand L.: "A practical guide for using statistical tests to assess randomized
rd
algorithms in software engineering". In Proceedings of the 33 International Conference on
Software Engineering, New York, NY, USA, May 2011, pp. 1-10.
[i.64] Pham H.V., Qian S., Wang J., Lutellier T., Rosenthal J., Tan L., Yu Y. & Nagappan, N., 2020,
December: "Problems and opportunities in training deep learning software systems: An analysis of
th
variance". In Proceedings of the 35 IEEE/ACM international conference on automated software
engineering (pp. 771-783).
[i.65] Elsken T., Metzen J.H. & Hutter F. (2019): "Neural Architecture Search: A Survey".
[i.66] Dehmer M. & Pickl S. (2015): "Network Complexity Measures: an Information-theoretic
Approach".
[i.67] M. Fan, W. Wei, X. Xie, Y. Liu, X. Guan and T. Liu. (2021): "Can We Trust Your Explanations?
TM
Sanity Checks for Interpreters in Android Malware Analysis". In IEEE Transactions on
Information Forensics and Security, vol. 16, pp. 838-853.
ETSI
10 ETSI TR 104 032 V1.1.1 (2024-02)
[i.68] Yan S., Tao G., Liu X. et al. (2020): "Correlations between deep neural network model coverage
th
criteria and model quality". In Proceedings of the 28 ACM Joint Meeting on European Software
Engineering Conference and Symposium on the Foundations of Software Engineering, New York,
NY, USA: Association for Computing Machinery, pp. 775-787.
[i.69] Lyu Y., Rajbahadur G. K., Lin D., Chen B. & Jiang Z. M. (2021): "Towards a consistent
interpretation of AIOps models". ACM Transactions on Software Engineering and Methodology
(TOSEM), 31(1), 1-38.
[i.70] Tulio Ribeiro M., Singh S. & Guestrin C.: "Why should I trust you?: Explaining the predictions of
any classifier". Proceedings of the 22nd ACM SIGKDD international conference on knowledge
discovery and data mining. ACM (2016).
[i.71] Tulio Ribeiro M., Singh S. & Guestrin C.:"Anchors: High-Precision Model-Agnostic
Explanations" AAAI Conference on Artificial Intelligence (AAAI), 2018.
[i.72] Lundberg, S. M. & Su-In, L. (2017): "A unified approach to interpreting model predictions.
Advances in Neural Information Processing Systems".
[i.73] Rajbahadur G. K., Wang S., Oliva G., Kamei Y, Hassan A. E. (2021): "The impact of feature
TM
importance methods on the interpretation of defect classifiers". IEEE Transactions on Software
Engineering, pp. 10.1109/TSE.2021.3056941.
3 Definition of terms, symbols and abbreviations
3.1 Terms
Void.
3.2 Symbols
Void.
3.3 Abbreviations
For the purposes of the present document, the following abbreviations apply:
AI Artificial Intelligence
AIA Artificial Intelligence Agent
AIH Artificial Intelligence Host
API Application Programming Interface
AUC Area Under the Curve
DevOps Development Operations
DNN Deep Neural Network(s)
DRM Digital Right Management
FL Federated Learning
GAN Generative Adversarial Network
GPU Graphics Processing Unit
IP Intellectual Property
IPR Intellectual Property Rights
ML Machine Learning
MLaaS Machine Learning as a Service
MLOps Machine Learning Operations
NDA Non Disclosure Agreement
OPM Open Provenance Model
OS Operating System
PKI Public Key Infrastructure
TEE Trusted Execution Environment
ETSI
11 ETSI TR 104 032 V1.1.1 (2024-02)
4 Introduction
4.1 The importance of traceability
During recent years, Machine Learning, and especially Deep Learning, put a new light on all kind of industries that
solve complex problems by providing noteworthy capabilities in domains such as voice recognition, image
comprehension, natural language processing, autonomous vehicles, and many more. However, its adoption is slowed
down by the difficulty of training of an appropriate model that would fit the particular use case. To train a good ML
model, it is indeed necessary to gather massive amount of relevant and ideally non-synthetic data, which can be a real
burden as data can be expensive to produce, classified or protected by privacy regulations. Moreover, a sophisticated
training requires also high computational resources, which represents a non-negligible cost. Finally, many engineers and
researchers may be enlisted to design, implement and optimize an advanced ML model. All of the above-mentioned
issues may discourage non AI-specialized actors from training its own models from scratch.
Two new challenges arise in such a context. First, new business models are being developed, aiming at monetizing data,
pre-trained models, MLaaS services, or the training procedure. This fuels the development of web scrappers and model
extraction attacks, as both data and models became attractive targets for attackers. Instead of spending time on acquiring
a dataset or training a new model, attackers may now try to steal those already created by their competitors. It becomes
thus crucial to ensure protection of ownership rights of the exposed models and data (besides, data theft may violate
privacy regulations). Second, as training of a model is a fastidious and costly process, its reproducibility becomes a
challenge on its own.
NOTE: The market for AI services is estimated to exceed 5,5 trillion dollars by 2027 [i.4].
Motivated by the growing complexity of modern AI and the development of new markets for ML models, the present
document addresses two aspects of ML traceability: traceability that aims at providing protection against model misuse
and traceability that improves AI trustworthiness. It focuses on techniques designed for Machine Learning and Deep
Learning models, as those were the most studied in the context of AI traceability.
4.2 Traceability for ownership rights protection and fight against
AI misuse
The ETSI Problem Statement [i.2] mentions reverse engineering and AI misuse as potential threats to AI systems.
Indeed, an attacker unable to produce its own ML model may try to extract an existing one from the competitor's system
in order to use it without paying or resell it. A similar situation may happen if an authorized user of a model decides to
make an unauthorized usage of a model. While the ETSI Mitigation Strategy Threats [i.3] provides with an overview of
some defence mechanisms against model extraction, the present document goes deeper into the problem. It presents
both classical Digital Right Management (DRM) and novel traceability techniques, such as ML watermarking, that can
be used to prevent or detect theft or misuse at any time of the ML lifecycle.
EXAMPLE 1: In 2020, a large social network operator introduced "radioactive data" to fight against datasets
thefts. It was a reaction to the rise of big data startups, which are scrapping publicly available
photographs from social networks [i.5].
EXAMPLE 2: Language models, such as the recently released "Chat GPT", can be used as universal writer's
assistant that can help completing scholar assignments. Watermarking their outputs may help
detect a misuse of the tool.
4.3 Traceability for trustworthy AI
Beside the challenge of protecting AI from theft or misuse, the present document tackles also the problem of traceability
that aims at guaranteeing an easier reproduction of results. Such traceability can be understood as the monitoring of the
whole lifecycle of a model, which includes not only the tracing of a model but also of its data and metadata as well as
the details of the training process. It aims at accelerating the AI lifecycle and fostering collaboration and model reuse. In
the context of secure AI, it should help detect attacks and enable easier integration of security mechanisms in the model
development lifecycle.
ETSI
12 ETSI TR 104 032 V1.1.1 (2024-02)
EXAMPLE: In a paper from Nature [i.6], AI was shown to outperform radiologists in specific settings on the
task of breast cancer screening. At the same time, it was pointed out that the absence of
sufficiently documented methods and computer code underlying the study effectively undermined
the scientific value of the research.
5 Traceability for ownership rights protection
5.1 Classical Digital Right Management
5.1.0 Introduction
This clause describes how classical DRM means - such as patents, trade secrets, and copyrights- can apply to AI and
tries to identify where are their limitations. It is based on information contained in [i.7].
5.1.1 Classical DRM mechanisms explained
Intellectual Property Rights (IPR) are legal rights that protect non-tangible business assets against various types of
misuse by third parties. Such misuse can be stopped by a legal injunction issued by a court, often combined with claims
of financial damages and/or seizure of infringing products. However, each type of IPR has its own particular
requirements and limitations. Below, copyrights, patents, database rights and trade secrets are presented.
NOTE: IP rights protection may vary depending on the country. For instance, many EU Member States do not
even see trade secrets as part of the IP domain. Moreover, there is no consensus over the definition of an
algorithm and the way it should be protected, e.g. algorithms may be protected in Italy but not in
Germany [i.8].
Copyright is the most well-known type of IPR. A copyright is the right to forbid copying and dissemination of a
protected work. Traditionally this right has been used much in the creative arts, e.g. for music, books and photographs.
However, copyright applies just as much to business works such as software, manuals, whitepapers, company videos
and so on.
Patents are the heavy lifters of the IPR world. When an innovation is protected by a patent, a patent owner may prevent
others from making, using or selling any device incorporating that innovation. Unlike a copyright, a patent protects any
independent re-creation. The patent holder can demand royalties or simply put an end to someone's commercial use of
his innovation. The major drawback is the application process, which involves costly fees and a multi-year examination
process with uncertain outcome. A complication with software is the case law on "software patents," which may be
perceived negatively in some countries. Thus, it may be difficult to enforce a patent on an innovation that heavily draws
on software or automation in all jurisdictions.
A relative newcomer in the IPR world is the database right. Introduced in Europe in the late 1990s, the database right
protects a collection of information against copying and reuse. The main requirement to qualify for a database right is
that substantial investment was made in the creation or maintenance of the data in the database. As with copyright, no
formal registration or application is required. Examples of protected databases include online dictionaries, labelled
image collections and source data for cartographical maps. In all cases, the data should be organized for search and
browsing. Outside of the European Union, however, the database right is not recognized, which further complicates
IPR. The U.S. has a long-standing legal tradition that collections of data are not protectable by IPR; only creative works
can be protected under copyright.
The status of trade secrets in the IPR world differs around the world, but in general, misappropriation of well-protected
information is actionable by law. In this instance, the owner of the information would be required to show how it
applied adequate security measures against unauthorized access. A would-be trade secret thief could then counter by
proving that the information was already available in the public domain.
Typically, companies guard their secrets by signing Non-Disclosure Agreements (NDAs) with customers or other third
parties. Strict contractual obligations then prohibit copying or reuse, in some jurisdictions strengthened by contractual
fines or other legal measures. NDA provisions may also be present in other agreements. However, someone who learns
the confidential data from a legitimate purchase of a product is not bound by such provisions, even when using special
techniques such as reverse engineering. This limits the strength of trade secret law.
ETSI
13 ETSI TR 104 032 V1.1.1 (2024-02)
5.1.2 Protection of the training set
Creating a good training set for a particular ML application can be a time-consuming and expensive effort and, although
in a typical setting an infringer has no direct access to this training set, the ability to copy the set is easy if the person
has access. This is where IP law comes in.
A training set would be protected with a database right, if the owner of the training set has its principal place of business
in the European Union. However, such right would only be enforceable against an infringer in the same jurisdiction.
Whether copyright can be claimed on an ML training set is a more difficult question. A training set is not created to be a
piece of art. The typical intention is to ensure the data fits the use case. Creating a well-fitting set of data on a topic may
not necessarily not be a creative activity under copyright law. One potential copyright claim are the data classification
descriptors.
EXAMPLE 1: If categories are chosen through a creative process - "beautiful/ugly", "strong/weak", "big/small" -
then the training set could be said to be protected by copyright through this creative labelling. A
classification based on factual elements - "cat/dog", "traffic light/ streetlight/parking sign" - does
not necessarily impart creativity and therefore possibly may not allow for copyright protection.
In some applications, training sets are generated by simulation or other artificial means. Arguably, these training sets
could be copyright protected as the choice of how to simulate or generate could be seen as a creative choice. However,
to date, this has not been challenged in court.
EXAMPLE 2: AI, and more particularly GANs, can be used to generated non-private training datasets from
private data. Such datasets will preserve the characteristics of the original datasets without leaking
sensitive information.
Companies will often consider their training sets to be carefully guarded secrets. Since a training set is not required to
be shared for the ML model to be used, it seems straightforward. The best approaches are to both guard the training set
from illicit copying and apply strong contractual restrictions to parties that have the training set.
5.1.3 Protection of the training parameters
The training set and model are only a part of the value of a good ML system. The parameters that steer the training
algorithm may also have value: choosing the right training parameters takes time and effort from highly trained
engineers.
For the set of training parameters that create the ML system, copyright protection is a sensible approach. If a data
scientist determining these parameters uses creative efforts to select the right training parameters, the resulting set of
parameters would likely be protected by copyright. But if the training parameters were found through exhaustive search
(e.g. evaluating a number of options proposed in the literature) or algorithmic process, copyright may not be available.
The same would apply to the model that is produced using those training parameters and a given training set.
A database right is least likely on the parameter set because one criterion for database rights is that it applies to a
collection of individual elements that are systematically or methodically arranged. A parameter set is unlikely to fit t
...