mlsanityData to 7 Oct 2026

Distillation-Resistant Watermarking for Model Protection in NLP

arXiv 2210.03312 ↗First posted 7 Oct 2022cs.CL cs.LGcited by 29

On the map: LLM safety and security → Watermarking generated content and models

Abstract (arXiv)

How can we protect the intellectual property of trained NLP models? Modern NLP models are prone to stealing by querying and distilling from their publicly exposed APIs. However, existing protection methods such as watermarking only work for images but are not applicable to text. We propose Distillation-Resistant Watermarking (DRW), a novel technique to protect NLP models from being stolen via distillation. DRW protects a model by injecting watermarks into the victim's prediction probability corresponding to a secret key and is able to detect such a key by probing a suspect model. We prove that a protected model still retains the original accuracy within a certain bound. We evaluate DRW on a diverse set of NLP tasks including text classification, part-of-speech tagging, and named entity recognition. Experiments show that DRW protects the original model and detects stealing suspects at 100% mean average precision for all four tasks while the prior method fails on two.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in Watermarking generated content and models, one of 12 directions in LLM safety and security. Builds on 22 of its 30 references that are in the map. First posted on arXiv 7 Oct 2022.

Builds on

22 of 30 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Cosine Model Watermarking Against Ensemble Distillation5 Mar
  2. 2021Copy, Right? A Testing Framework for Copyright Protection of Deep Learning Models10 Dec
  3. 2021Protecting Intellectual Property of Language Generation APIs with Lexical Watermark5 Dec

Other references · 19

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2016Stealing Machine Learning Models via Prediction APIscited by 602
  7. 2018Turning Your Weakness Into a Strength: Watermarking Deep Neural Networks by Backdooringcited by 334
  8. 2018Knockoff Nets: Stealing Functionality of Black-Box Modelscited by 285
  9. 2017Embedding Watermarks into Deep Neural Networkscited by 227
  10. 2018PRADA: Protecting against DNN Model Stealing Attackscited by 159
  11. 2021Towards Understanding Knowledge Distillationcited by 158
  12. 2020Entangled Watermarks as a Defense against Model Extractioncited by 144
7 more
  1. 2019Thieves on Sesame Street! Model Extraction of BERT-based APIscited by 136
  2. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 116
  3. 2017Adversarial Frontier Stitching for Remote Neural Network Watermarkingcited by 110
  4. 2019DAWN: Dynamic Adversarial Watermarking of Neural Networkscited by 94
  5. 2020Imitation Attacks and Defenses for Black-box Machine Translation Systemscited by 78
  6. 2018Copycat CNN: Stealing Knowledge by Persuading Confession with Random Non-Labeled Datacited by 69
  7. 2021Model Extraction and Adversarial Transferability, Your BERT is Vulnerable!cited by 51

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 10; 2025: 7; 2026: 5.

29 papers.

Sort:

By area:

  1. 2023A Survey of Text Watermarking in the Era of Large Language Modelscited by 102
  2. 2023A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directionscited by 101
  3. 2023Protecting Language Generation Models via Invisible Watermarkingcited by 81
  4. 2024Watermark Stealing in Large Language Modelscited by 79
  5. 2024Detecting Multimedia Generated by Large AI Models: A Surveycited by 73
  6. 2023On the Learnability of Watermarks for Language Modelscited by 72
  7. 2023A Survey on Detection of LLMs-Generated Contentcited by 70
  8. 2024Instructional Fingerprinting of Large Language Modelscited by 64
  9. 2024SoK: Watermarking for AI-Generated Contentcited by 43
  10. 2024Threads of Subtlety: Detecting Machine-Generated Texts Through Discourse Motifscited by 18
  11. 2023Deep Intellectual Property Protection: A Surveycited by 16
  12. 2023Source Attribution for Large Language Model-Generated Datacited by 11
17 more
  1. 2025Scalable Fingerprinting of Large Language Modelscited by 11
  2. 2025Invisible Tokens, Visible Bills: The Urgent Need to Audit Hidden Operations in Opaque LLM Servicescited by 10
  3. 2025Position: LLM Watermarking Should Align Stakeholders' Incentives for Practical Adoptioncited by 7
  4. 2024Watermarking Generative Tabular Datacited by 6
  5. 2026Protecting Language Models Against Unauthorized Distillation through Trace Rewritingcited by 6
  6. 2024"Yes, My LoRD." Guiding Language Model Extraction with Locality Reinforced Distillationcited by 5
  7. 2025SAEMark: Steering Personalized Multilingual LLM Watermarks with Sparse Autoencoderscited by 5
  8. 2025Beyond Artificial Misalignment: Detecting and Grounding Semantic-Coordinated Multimodal Manipulationscited by 4
  9. 2025Data Provenance Auditing of Fine-Tuned Large Language Models with a Text-Preserving Techniquecited by 3
  10. 2025Distributional Statistics Restore Training Data Auditability in One-step Distilled Diffusion Modelscited by 2
  11. 2026RedAct: Redacting Agent Capability Traces for Procedural Skill Protectioncited by 2
  12. 2026Let Them Steal: Trapping Large Language Model Extraction Attacks with Knowledge Honeypotcited by 1
  13. 2024Vietnamese AI Generated Text Detectioncited by 0
  14. 2024Protecting Copyright of Medical Pre-trained Language Models: Training-Free Backdoor Model Watermarkingcited by 0
  15. 2024Signal Watermark on Large Language Modelscited by 0
  16. 2026Answer-then-Edit: Reasoning Skeleton Editing for Anti-Distillation with Preserved Utilitycited by 0
  17. 2026Steal the Knowledge, Inherit the Mark: TwinMark for Distillation Watermarking via Second Moments and Class Multiplexingcited by 0