# A Watermark for Order-Agnostic Language Models

arXiv: https://arxiv.org/abs/2410.13805 · HTML: https://mlsanity.com/paper/2410.13805/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Watermarking generated content and models, one of 12 directions in LLM safety and security.
- Builds on 21 of its 45 references that are in the map.
- First posted on arXiv 17 Oct 2024.

## Placed in

- LLM safety and security / [Watermarking generated content and models](https://mlsanity.com/safety/watermarking-generated-content-and-models/)

## Builds on (most cited first, 21 of 21)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [A Watermark for Large Language Models](https://mlsanity.com/paper/2301.10226/) (2023)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)
- [Robust Distortion-free Watermarks for Language Models](https://mlsanity.com/paper/2307.15593/) (2023)
- [Provable Robust Watermarking for AI-Generated Text](https://mlsanity.com/paper/2306.17439/) (2023)
- [Undetectable Watermarks for Language Models](https://mlsanity.com/paper/2306.09194/) (2023)
- [On the Reliability of Watermarks for Large Language Models](https://mlsanity.com/paper/2306.04634/) (2023)
- [A Semantic Invariant Robust Watermark for Large Language Models](https://mlsanity.com/paper/2310.06356/) (2023)
- [Unbiased Watermark for Large Language Models](https://mlsanity.com/paper/2310.10669/) (2023)
- [REMARK-LLM: A Robust and Efficient Watermarking Framework for Generative Large Language Models](https://mlsanity.com/paper/2310.12362/) (2023)
- [An Unforgeable Publicly Verifiable Watermark for Large Language Models](https://mlsanity.com/paper/2307.16230/) (2023)
- [Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selection](https://mlsanity.com/paper/2402.12501/) (2024)
- [Token-Specific Watermarking with Enhanced Detectability and Semantic Coherence for Large Language Models](https://mlsanity.com/paper/2402.18059/) (2024)
- [Adversarial Weight Perturbation Improves Generalization in Graph Neural Networks](https://mlsanity.com/paper/2212.04983/) (2022)
- [Learning to Learn from APIs: Black-Box Data-Free Meta-Learning](https://mlsanity.com/paper/2305.18413/) (2023)
- [RetrievalGuard: Provably Robust 1-Nearest Neighbor Image Retrieval](https://mlsanity.com/paper/2206.11225/) (2022)
- [Few-Shot Class Incremental Learning with Attention-Aware Self-Adaptive Prompt](https://mlsanity.com/paper/2403.09857/) (2024)
- [Distortion-free Watermarks are not Truly Distortion-free under Watermark Key Collisions](https://mlsanity.com/paper/2406.02603/) (2024)
- [A Law of Robustness beyond Isoperimetry](https://mlsanity.com/paper/2202.11592/) (2022)
- [NAST: Non-Autoregressive Spatial-Temporal Transformer for Time Series Forecasting](https://mlsanity.com/paper/2102.05624/) (2021)

## Built on by (most cited first, 14 of 14)

- [AdaDetectGPT: Adaptive Detection of LLM-Generated Text with Statistical Guarantees](https://mlsanity.com/paper/2510.01268/) (2025)
- [Improved Unbiased Watermark for Large Language Models](https://mlsanity.com/paper/2502.11268/) (2025)
- [In-Context Watermarks for Large Language Models](https://mlsanity.com/paper/2505.16934/) (2025)
- [Watermarking Diffusion Language Models](https://mlsanity.com/paper/2509.24368/) (2025)
- [Toward Robust Non-Transferable Learning: A Survey and Benchmark](https://mlsanity.com/paper/2502.13593/) (2025)
- [Explore Data Left Behind in Reinforcement Learning for Reasoning Language Models](https://mlsanity.com/paper/2511.04800/) (2025)
- [Detecting Post-generation Edits to Watermarked LLM Outputs via Combinatorial Watermarking](https://mlsanity.com/paper/2510.01637/) (2025)
- [More Haste, Less Speed: Weaker Single-Layer Watermark Improves Distortion-Free Watermark Ensembles](https://mlsanity.com/paper/2602.11793/) (2026)
- [SAMark: A Self-Anchored Text Watermarking with Paragraph-Level Paraphrase Robustness](https://mlsanity.com/paper/2605.25796/) (2026)
- [Chainwash: Multi-Step Rewriting Attacks on Diffusion Language Model Watermarks](https://mlsanity.com/paper/2605.05503/) (2026)
- [AliMark: Enhancing Robustness of Sentence-Level Watermarking Against Text Paraphrasing](https://mlsanity.com/paper/2605.29434/) (2026)
- [SAC-Copula: Quality-Preserving Watermarking for Diffusion Language Models via Smooth Correlated Gumbel Fields](https://mlsanity.com/paper/2608.20839/) (2026)
- [DenMark: Robust Semantic Watermarking for Diffusion Language Models](https://mlsanity.com/paper/2609.14257/) (2026)
- [Anchor-ECC: Local Integrity Checking for Watermarked LLM Outputs via Error-Correcting Codes](https://mlsanity.com/paper/2609.38722/) (2026)

## Abstract (arXiv)

Statistical watermarking techniques are well-established for sequentially decoded language models (LMs). However, these techniques cannot be directly applied to order-agnostic LMs, as the tokens in order-agnostic LMs are not generated sequentially. In this work, we introduce Pattern-mark, a pattern-based watermarking framework specifically designed for order-agnostic LMs. We develop a Markov-chain-based watermark generator that produces watermark key sequences with high-frequency key patterns. Correspondingly, we propose a statistical pattern-based detection algorithm that recovers the key sequence during detection and conducts statistical tests based on the count of high-frequency patterns. Our extensive evaluations on order-agnostic LMs, such as ProteinMPNN and CMLM, demonstrate Pattern-mark's enhanced detection efficiency, generation quality, and robustness, positioning it as a superior watermarking technique for order-agnostic LMs.