# Task-Agnostic Language Model Watermarking via High Entropy Passthrough Layers

arXiv: https://arxiv.org/abs/2412.12563 · HTML: https://mlsanity.com/paper/2412.12563/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Watermarking generated content and models, one of 12 directions in LLM safety and security.
- Builds on 25 of its 46 references that are in the map.
- First posted on arXiv 17 Dec 2024.

## Placed in

- LLM safety and security / [Watermarking generated content and models](https://mlsanity.com/safety/watermarking-generated-content-and-models/)

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [A Watermark for Large Language Models](https://mlsanity.com/paper/2301.10226/) (2023)
- [PAWS: Paraphrase Adversaries from Word Scrambling](https://mlsanity.com/paper/1904.01130/) (2019)
- [Turning Your Weakness Into a Strength: Watermarking Deep Neural Networks by Backdooring](https://mlsanity.com/paper/1802.04633/) (2018)
- [The Stable Signature: Rooting Watermarks in Latent Diffusion Models](https://mlsanity.com/paper/2303.15435/) (2023)
- [Poisoning Language Models During Instruction Tuning](https://mlsanity.com/paper/2305.00944/) (2023)
- [Simple linear attention language models balance the recall-throughput tradeoff](https://mlsanity.com/paper/2402.18668/) (2024)
- [On the Reliability of Watermarks for Large Language Models](https://mlsanity.com/paper/2306.04634/) (2023)
- [A Recipe for Watermarking Diffusion Models](https://mlsanity.com/paper/2303.10137/) (2023)
- [Watermarking Images in Self-Supervised Latent Spaces](https://mlsanity.com/paper/2112.09581/) (2021)
- [CATER: Intellectual Property Protection on Text Generation APIs via Conditional Watermarks](https://mlsanity.com/paper/2209.08773/) (2022)
- [IPGuard: Protecting Intellectual Property of Deep Neural Networks via Fingerprinting the Classification Boundary](https://mlsanity.com/paper/1910.12903/) (2019)
- [Robust Watermarking of Neural Network with Exponential Weighting](https://mlsanity.com/paper/1901.06151/) (2019)
- [Watermarking Pre-trained Language Models with Backdooring](https://mlsanity.com/paper/2210.07543/) (2022)
- [Backdoor Learning on Sequence to Sequence Models](https://mlsanity.com/paper/2305.02424/) (2023)
- [DeepSigns: A Generic Watermarking Framework for IP Protection of Deep Learning Models](https://mlsanity.com/paper/1804.00750/) (2018)
- [Concealed Data Poisoning Attacks on NLP Models](https://mlsanity.com/paper/2010.12563/) (2020)
- [Piracy Resistant Watermarks for Deep Neural Networks](https://mlsanity.com/paper/1910.01226/) (2019)
- [Protecting Your NLG Models with Semantic and Robust Watermarks](https://mlsanity.com/paper/2112.05428/) (2021)
- [Robust Black-box Watermarking for Deep NeuralNetwork using Inverse Document Frequency](https://mlsanity.com/paper/2103.05590/) (2021)

## Built on by (most cited first, 1 of 1)

- [DERMARK: A Dynamic, Efficient and Robust Multi-bit Watermark for Large Language Models](https://mlsanity.com/paper/2502.05213/) (2025)

## Abstract (arXiv)

In the era of costly pre-training of large language models, ensuring the intellectual property rights of model owners, and insuring that said models are responsibly deployed, is becoming increasingly important. To this end, we propose model watermarking via passthrough layers, which are added to existing pre-trained networks and trained using a self-supervised loss such that the model produces high-entropy output when prompted with a unique private key, and acts normally otherwise. Unlike existing model watermarking methods, our method is fully task-agnostic, and can be applied to both classification and sequence-to-sequence tasks without requiring advanced access to downstream fine-tuning datasets. We evaluate the proposed passthrough layers on a wide range of downstream tasks, and show experimentally our watermarking method achieves a near-perfect watermark extraction accuracy and false-positive rate in most cases without damaging original model performance. Additionally, we show our method is robust to both downstream fine-tuning, fine-pruning, and layer removal attacks, and can be trained in a fraction of the time required to train the original model. Code is available in the paper.