# Cosine Model Watermarking Against Ensemble Distillation

arXiv: https://arxiv.org/abs/2203.02777 · HTML: https://mlsanity.com/paper/2203.02777/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Watermarking generated content and models, one of 12 directions in LLM safety and security.
- Builds on 17 of its 27 references that are in the map.
- First posted on arXiv 5 Mar 2022.

## Placed in

- LLM safety and security / [Watermarking generated content and models](https://mlsanity.com/safety/watermarking-generated-content-and-models/)

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Practical Black-Box Attacks against Machine Learning](https://mlsanity.com/paper/1602.02697/) (2016)
- [Stealing Machine Learning Models via Prediction APIs](https://mlsanity.com/paper/1609.02943/) (2016)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [Turning Your Weakness Into a Strength: Watermarking Deep Neural Networks by Backdooring](https://mlsanity.com/paper/1802.04633/) (2018)
- [Knockoff Nets: Stealing Functionality of Black-Box Models](https://mlsanity.com/paper/1812.02766/) (2018)
- [Embedding Watermarks into Deep Neural Networks](https://mlsanity.com/paper/1701.04082/) (2017)
- [High Accuracy and High Fidelity Extraction of Neural Networks](https://mlsanity.com/paper/1909.01838/) (2019)
- [PRADA: Protecting against DNN Model Stealing Attacks](https://mlsanity.com/paper/1805.02628/) (2018)
- [Entangled Watermarks as a Defense against Model Extraction](https://mlsanity.com/paper/2002.12200/) (2020)
- [Adversarial Frontier Stitching for Remote Neural Network Watermarking](https://mlsanity.com/paper/1711.01894/) (2017)
- [DAWN: Dynamic Adversarial Watermarking of Neural Networks](https://mlsanity.com/paper/1906.00830/) (2019)
- [Deep Neural Network Fingerprinting by Conferrable Adversarial Examples](https://mlsanity.com/paper/1912.00888/) (2019)
- [MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Tricks](https://mlsanity.com/paper/2009.08453/) (2020)
- [Unifying Heterogeneous Classifiers with Distillation](https://mlsanity.com/paper/1904.06062/) (2019)

## Built on by (most cited first, 13 of 13)

- [Protecting Language Generation Models via Invisible Watermarking](https://mlsanity.com/paper/2302.03162/) (2023)
- [Distillation-Resistant Watermarking for Model Protection in NLP](https://mlsanity.com/paper/2210.03312/) (2022)
- [Watermarking Techniques for Large Language Models: A Survey](https://mlsanity.com/paper/2409.00089/) (2024)
- [Isolation and Induction: Training Robust Deep Neural Networks against Model Stealing Attacks](https://mlsanity.com/paper/2308.00958/) (2023)
- [DeepEclipse: How to Break White-Box DNN-Watermarking Schemes](https://mlsanity.com/paper/2403.03590/) (2024)
- [MAP: MAsk-Pruning for Source-Free Model Intellectual Property Protection](https://mlsanity.com/paper/2403.04149/) (2024)
- [Towards Open Federated Learning Platforms: Survey and Vision from Technical and Legal Perspectives](https://mlsanity.com/paper/2307.02140/) (2023)
- [Performance-lossless Black-box Model Watermarking](https://mlsanity.com/paper/2312.06488/) (2023)
- [Persistence of Backdoor-based Watermarks for Neural Networks: A Comprehensive Evaluation](https://mlsanity.com/paper/2501.02704/) (2025)
- [Customized Watermarking for Deep Neural Networks via Label Distribution Perturbation](https://mlsanity.com/paper/2208.05477/) (2022)
- [WGLE:Backdoor-free and Multi-bit Black-box Watermarking for Graph Neural Networks](https://mlsanity.com/paper/2506.08602/) (2025)
- [From Zero to Hero: Detecting Leaked Data through Synthetic Data Injection and Model Querying](https://mlsanity.com/paper/2310.04145/) (2023)
- [Steal the Knowledge, Inherit the Mark: TwinMark for Distillation Watermarking via Second Moments and Class Multiplexing](https://mlsanity.com/paper/2609.19011/) (2026)

## Abstract (arXiv)

Many model watermarking methods have been developed to prevent valuable deployed commercial models from being stealthily stolen by model distillations. However, watermarks produced by most existing model watermarking methods can be easily evaded by ensemble distillation, because averaging the outputs of multiple ensembled models can significantly reduce or even erase the watermarks. In this paper, we focus on tackling the challenging task of defending against ensemble distillation. We propose a novel watermarking technique named CosWM to achieve outstanding model watermarking performance against ensemble distillation. CosWM is not only elegant in design, but also comes with desirable theoretical guarantees. Our extensive experiments on public data sets demonstrate the excellent performance of CosWM and its advantages over the state-of-the-art baselines.