# Functional Invariants to Watermark Large Transformers

arXiv: https://arxiv.org/abs/2310.11446 · HTML: https://mlsanity.com/paper/2310.11446/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 17 Oct 2023.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [Turning Your Weakness Into a Strength: Watermarking Deep Neural Networks by Backdooring](https://mlsanity.com/paper/1802.04633/) (2018)
- [The Impact of Positional Encoding on Length Generalization in Transformers](https://mlsanity.com/paper/2305.19466/) (2023)
- [The Stable Signature: Rooting Watermarks in Latent Diffusion Models](https://mlsanity.com/paper/2303.15435/) (2023)
- [Embedding Watermarks into Deep Neural Networks](https://mlsanity.com/paper/1701.04082/) (2017)
- [[Extended version] Rethinking Deep Neural Network Ownership Verification: Embedding Passports to Defeat Ambiguity Attacks](https://mlsanity.com/paper/1909.07830/) (2019)
- [WOUAF: Weight Modulation for User Attribution and Fingerprinting in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2306.04744/) (2023)
- [Protecting Intellectual Property of Generative Adversarial Networks from Ambiguity Attack](https://mlsanity.com/paper/2102.04362/) (2021)
- [Passport-aware Normalization for Deep Model Protection](https://mlsanity.com/paper/2010.15824/) (2020)
- [Protect, Show, Attend and Tell: Empowering Image Captioning Models with Ownership Protection](https://mlsanity.com/paper/2008.11009/) (2020)
- [Rethinking White-Box Watermarks on Deep Learning Models under Neural Structural Obfuscation](https://mlsanity.com/paper/2303.09732/) (2023)
- [ROSE: A RObust and SEcure DNN Watermarking](https://mlsanity.com/paper/2206.11024/) (2022)

## Built on by (most cited first, 15 of 15)

- [Hey, That's My Model! Introducing Chain & Hash, An LLM Fingerprinting Technique](https://mlsanity.com/paper/2407.10887/) (2024)
- [REEF: Representation Encoding Fingerprints for Large Language Models](https://mlsanity.com/paper/2410.14273/) (2024)
- [Watermarking Techniques for Large Language Models: A Survey](https://mlsanity.com/paper/2409.00089/) (2024)
- [Copyright Protection for Large Language Models: A Survey of Methods, Challenges, and Trends](https://mlsanity.com/paper/2508.11548/) (2025)
- [MergePrint: Merge-Resistant Fingerprints for Robust Black-box Ownership Verification of Large Language Models](https://mlsanity.com/paper/2410.08604/) (2024)
- [Position: LLM Watermarking Should Align Stakeholders' Incentives for Practical Adoption](https://mlsanity.com/paper/2510.18333/) (2025)
- [AttnDiff: Attention-based Differential Fingerprinting for Large Language Models](https://mlsanity.com/paper/2604.05502/) (2026)
- [Revisiting Black-Box Model Ownership Verification through Information Theory](https://mlsanity.com/paper/2409.06130/) (2024)
- [AWM: Accurate Weight-Matrix Fingerprint for Large Language Models](https://mlsanity.com/paper/2510.06738/) (2025)
- [Towards the Resistance of Neural Network Watermarking to Fine-tuning](https://mlsanity.com/paper/2505.01007/) (2025)
- [SLAM: Structural Linguistic Activation Marking for Language Models](https://mlsanity.com/paper/2605.05443/) (2026)
- [Invariant-based Robust Weights Watermark for Large Language Models](https://mlsanity.com/paper/2507.08288/) (2025)
- [Protecting Copyright of Medical Pre-trained Language Models: Training-Free Backdoor Model Watermarking](https://mlsanity.com/paper/2409.10570/) (2024)
- [FNF: Functional Network Fingerprint for Large Language Models](https://mlsanity.com/paper/2601.22692/) (2026)
- [Permutation-Based Stegomalware in Large Language Models: Threats and Countermeasures](https://mlsanity.com/paper/2609.16193/) (2026)

## Abstract (arXiv)

The rapid growth of transformer-based models increases the concerns about their integrity and ownership insurance. Watermarking addresses this issue by embedding a unique identifier into the model, while preserving its performance. However, most existing approaches require to optimize the weights to imprint the watermark signal, which is not suitable at scale due to the computational cost. This paper explores watermarks with virtually no computational cost, applicable to a non-blind white-box setting (assuming access to both the original and watermarked networks). They generate functionally equivalent copies by leveraging the models' invariance, via operations like dimension permutations or scaling/unscaling. This enables to watermark models without any change in their outputs and remains stealthy. Experiments demonstrate the effectiveness of the approach and its robustness against various model transformations (fine-tuning, quantization, pruning), making it a practical solution to protect the integrity of large models.