# Distillation-Resistant Watermarking for Model Protection in NLP

arXiv: https://arxiv.org/abs/2210.03312 · HTML: https://mlsanity.com/paper/2210.03312/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Watermarking generated content and models, one of 12 directions in LLM safety and security.
- Builds on 22 of its 30 references that are in the map.
- First posted on arXiv 7 Oct 2022.

## Placed in

- LLM safety and security / [Watermarking generated content and models](https://mlsanity.com/safety/watermarking-generated-content-and-models/)

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Stealing Machine Learning Models via Prediction APIs](https://mlsanity.com/paper/1609.02943/) (2016)
- [Turning Your Weakness Into a Strength: Watermarking Deep Neural Networks by Backdooring](https://mlsanity.com/paper/1802.04633/) (2018)
- [Knockoff Nets: Stealing Functionality of Black-Box Models](https://mlsanity.com/paper/1812.02766/) (2018)
- [Embedding Watermarks into Deep Neural Networks](https://mlsanity.com/paper/1701.04082/) (2017)
- [PRADA: Protecting against DNN Model Stealing Attacks](https://mlsanity.com/paper/1805.02628/) (2018)
- [Towards Understanding Knowledge Distillation](https://mlsanity.com/paper/2105.13093/) (2021)
- [Entangled Watermarks as a Defense against Model Extraction](https://mlsanity.com/paper/2002.12200/) (2020)
- [Thieves on Sesame Street! Model Extraction of BERT-based APIs](https://mlsanity.com/paper/1910.12366/) (2019)
- [Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective](https://mlsanity.com/paper/2102.00650/) (2021)
- [Adversarial Frontier Stitching for Remote Neural Network Watermarking](https://mlsanity.com/paper/1711.01894/) (2017)
- [DAWN: Dynamic Adversarial Watermarking of Neural Networks](https://mlsanity.com/paper/1906.00830/) (2019)
- [Protecting Intellectual Property of Language Generation APIs with Lexical Watermark](https://mlsanity.com/paper/2112.02701/) (2021)
- [Imitation Attacks and Defenses for Black-box Machine Translation Systems](https://mlsanity.com/paper/2004.15015/) (2020)
- [Copycat CNN: Stealing Knowledge by Persuading Confession with Random Non-Labeled Data](https://mlsanity.com/paper/1806.05476/) (2018)
- [Model Extraction and Adversarial Transferability, Your BERT is Vulnerable!](https://mlsanity.com/paper/2103.10013/) (2021)
- [Copy, Right? A Testing Framework for Copyright Protection of Deep Learning Models](https://mlsanity.com/paper/2112.05588/) (2021)
- [Cosine Model Watermarking Against Ensemble Distillation](https://mlsanity.com/paper/2203.02777/) (2022)

## Built on by (most cited first, 30 of 30)

- [A Survey of Text Watermarking in the Era of Large Language Models](https://mlsanity.com/paper/2312.07913/) (2023)
- [A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions](https://mlsanity.com/paper/2310.14724/) (2023)
- [Watermark Stealing in Large Language Models](https://mlsanity.com/paper/2402.19361/) (2024)
- [Protecting Language Generation Models via Invisible Watermarking](https://mlsanity.com/paper/2302.03162/) (2023)
- [On the Learnability of Watermarks for Language Models](https://mlsanity.com/paper/2312.04469/) (2023)
- [Detecting Multimedia Generated by Large AI Models: A Survey](https://mlsanity.com/paper/2402.00045/) (2024)
- [A Survey on Detection of LLMs-Generated Content](https://mlsanity.com/paper/2310.15654/) (2023)
- [Instructional Fingerprinting of Large Language Models](https://mlsanity.com/paper/2401.12255/) (2024)
- [SoK: Watermarking for AI-Generated Content](https://mlsanity.com/paper/2411.18479/) (2024)
- [Threads of Subtlety: Detecting Machine-Generated Texts Through Discourse Motifs](https://mlsanity.com/paper/2402.10586/) (2024)
- [Deep Intellectual Property Protection: A Survey](https://mlsanity.com/paper/2304.14613/) (2023)
- [Scalable Fingerprinting of Large Language Models](https://mlsanity.com/paper/2502.07760/) (2025)
- [Source Attribution for Large Language Model-Generated Data](https://mlsanity.com/paper/2310.00646/) (2023)
- [Invisible Tokens, Visible Bills: The Urgent Need to Audit Hidden Operations in Opaque LLM Services](https://mlsanity.com/paper/2505.18471/) (2025)
- [Watermarking Generative Tabular Data](https://mlsanity.com/paper/2405.14018/) (2024)
- [SAEMark: Steering Personalized Multilingual LLM Watermarks with Sparse Autoencoders](https://mlsanity.com/paper/2508.08211/) (2025)
- [Position: LLM Watermarking Should Align Stakeholders' Incentives for Practical Adoption](https://mlsanity.com/paper/2510.18333/) (2025)
- [Protecting Language Models Against Unauthorized Distillation through Trace Rewriting](https://mlsanity.com/paper/2602.15143/) (2026)
- ["Yes, My LoRD." Guiding Language Model Extraction with Locality Reinforced Distillation](https://mlsanity.com/paper/2409.02718/) (2024)
- [Beyond Artificial Misalignment: Detecting and Grounding Semantic-Coordinated Multimodal Manipulations](https://mlsanity.com/paper/2509.12653/) (2025)
- [Data Provenance Auditing of Fine-Tuned Large Language Models with a Text-Preserving Technique](https://mlsanity.com/paper/2510.09655/) (2025)
- [RedAct: Redacting Agent Capability Traces for Procedural Skill Protection](https://mlsanity.com/paper/2606.10813/) (2026)
- [Let Them Steal: Trapping Large Language Model Extraction Attacks with Knowledge Honeypot](https://mlsanity.com/paper/2606.15810/) (2026)
- [Distributional Statistics Restore Training Data Auditability in One-step Distilled Diffusion Models](https://mlsanity.com/paper/2502.02970/) (2025)
- [Signal Watermark on Large Language Models](https://mlsanity.com/paper/2410.06545/) (2024)
- [BlackCATT: Black-box Collusion Aware Traitor Tracing in Federated Learning](https://mlsanity.com/paper/2602.12138/) (2026)
- [Vietnamese AI Generated Text Detection](https://mlsanity.com/paper/2405.03206/) (2024)
- [Protecting Copyright of Medical Pre-trained Language Models: Training-Free Backdoor Model Watermarking](https://mlsanity.com/paper/2409.10570/) (2024)
- [Answer-then-Edit: Reasoning Skeleton Editing for Anti-Distillation with Preserved Utility](https://mlsanity.com/paper/2607.20440/) (2026)
- [Steal the Knowledge, Inherit the Mark: TwinMark for Distillation Watermarking via Second Moments and Class Multiplexing](https://mlsanity.com/paper/2609.19011/) (2026)

## Abstract (arXiv)

How can we protect the intellectual property of trained NLP models? Modern NLP models are prone to stealing by querying and distilling from their publicly exposed APIs. However, existing protection methods such as watermarking only work for images but are not applicable to text. We propose Distillation-Resistant Watermarking (DRW), a novel technique to protect NLP models from being stolen via distillation. DRW protects a model by injecting watermarks into the victim's prediction probability corresponding to a secret key and is able to detect such a key by probing a suspect model. We prove that a protected model still retains the original accuracy within a certain bound. We evaluate DRW on a diverse set of NLP tasks including text classification, part-of-speech tagging, and named entity recognition. Experiments show that DRW protects the original model and detects stealing suspects at 100% mean average precision for all four tasks while the prior method fails on two.