mlsanityData to 14 Oct 2026

On the Reliability of Watermarks for Large Language Models

arXiv 2306.04634 ↗First posted 7 Jun 2023cs.LG cs.CL cs.CRcited by 189

On the map: LLM safety and security → Watermarking generated content and models

Abstract (arXiv)

As LLMs become commonplace, machine-generated text has the potential to flood the internet with spam, social media bots, and valueless content. Watermarking is a simple and effective strategy for mitigating such harms by enabling the detection and documentation of LLM-generated text. Yet a crucial question remains: How reliable is watermarking in realistic settings in the wild? There, watermarked text may be modified to suit a user's needs, or entirely rewritten to avoid detection. We study the robustness of watermarked text after it is re-written by humans, paraphrased by a non-watermarked LLM, or mixed into a longer hand-written document. We find that watermarks remain detectable even after human and machine paraphrasing. While these attacks dilute the strength of the watermark, paraphrases are statistically likely to leak n-grams or even longer fragments of the original text, resulting in high-confidence detections when enough tokens are observed. For example, after strong human paraphrasing the watermark is detectable after observing 800 tokens on average, when setting a 1e-5 false positive rate. We also consider a range of new detection schemes that are sensitive to short spans of watermarked text embedded inside a large document, and we compare the robustness of watermarking to other kinds of detectors.

Key facts. Cited by 189 papers in the map (data through 14 Oct 2026). Placed in Watermarking generated content and models, one of 12 directions in LLM safety and security. Builds on 38 of its 75 references that are in the map. First posted on arXiv 7 Jun 2023.

Builds on

38 of 75 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Who Wrote this Code? Watermarking for Code Generation24 May
  2. 2023Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense23 Mar
  3. 2023Can AI-Generated Text be Reliably Detected?17 Mar
  4. 2023A Watermark for Large Language Models24 Jan
  5. 2022CATER: Intellectual Property Protection on Text Generation APIs via Conditional Watermarks19 Sep
  6. 2022The Ethical Need for Watermarks in Machine-Generated Language7 Sep
  7. 2021Protecting Intellectual Property of Language Generation APIs with Lexical Watermark5 Dec

Other references · 31

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  8. 2023DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvaturecited by 726
  9. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  10. 2019Release Strategies and the Social Impacts of Language Modelscited by 513
  11. 2019Neural Text Generation with Unlikelihood Trainingcited by 503
  12. 2019GLTR: Statistical Detection and Visualization of Generated Textcited by 496
19 more
  1. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
  2. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 356
  3. 2022A Contrastive Framework for Neural Text Generationcited by 225
  4. 2023GPT detectors are biased against non-native English writerscited by 178
  5. 2020Automatic Detection of Machine Generated Text: A Critical Surveycited by 166
  6. 2020Adversarial Watermarking Transformer: Towards Tracing Text Provenance with Data Hidingcited by 130
  7. 2023The Science of Detecting LLM-Generated Textscited by 128
  8. 2020TweepFake: about Detecting Deepfake Tweetscited by 117
  9. 2019Real or Fake? Learning to Discriminate Machine from Human Generated Textcited by 99
  10. 2023On the Possibilities of AI-Generated Text Detectioncited by 88
  11. 2022Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methodscited by 79
  12. 2019Neural Linguistic Steganographycited by 78
  13. 2017Generating Steganographic Text with LSTMscited by 44
  14. 2020Neural Deepfake Detection with Factual Structure of Textcited by 41
  15. 2020Attacking Neural Text Detectorscited by 39
  16. 2019Towards Near-imperceptible Steganographic Textcited by 32
  17. 2021Unsupervised and Distributional Detection of Machine-Generated Textcited by 23
  18. 2021Paraphrastic Representations at Scalecited by 21
  19. 2020Limits of Detecting Text Generated by Large-Scale Language Modelscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 32; 2024: 54; 2025: 55; 2026: 48.

189 papers.

Sort:

By area:

  1. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  2. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  3. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  4. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  5. 2023Challenges and Applications of Large Language Modelscited by 254
  6. 2023Self-Consuming Generative Models Go MADcited by 250
  7. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 229
  8. 2023Factuality Challenges in the Era of Large Language Modelscited by 126
  9. 2024Secret Collusion among AI Agents: Multi-Agent Deception via Steganographycited by 113
  10. 2023A Semantic Invariant Robust Watermark for Large Language Modelscited by 112
  11. 2023A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directionscited by 105
  12. 2023NEFTune: Noisy Embeddings Improve Instruction Finetuningcited by 100
38 more
  1. 2024Watermark Stealing in Large Language Modelscited by 92
  2. 2023Identifying and Mitigating the Security Risks of Generative AIcited by 91
  3. 2023Watermarks in the Sand: Impossibility of Strong Watermarking for Generative Modelscited by 87
  4. 2023Three Bricks to Consolidate Watermarks for Large Language Modelscited by 84
  5. 2023On the Learnability of Watermarks for Language Modelscited by 81
  6. 2023REMARK-LLM: A Robust and Efficient Watermarking Framework for Generative Large Language Modelscited by 78
  7. 2024Adaptive Text Watermark for Large Language Modelscited by 67
  8. 2023Preventing Language Models From Hiding Their Reasoningcited by 65
  9. 2024Watermarking Makes Language Models Radioactivecited by 58
  10. 2023Advancing Beyond Identification: Multi-bit Watermark for Large Language Modelscited by 57
  11. 2023Towards Codable Watermarking for Injecting Multi-bits Information to LLMscited by 54
  12. 2024WaterMax: breaking the LLM watermark detectability-robustness-quality trade-offcited by 54
  13. 2024SoK: Watermarking for AI-Generated Contentcited by 51
  14. 2024No Free Lunch in LLM Watermarking: Trade-offs in Watermarking Design Choicescited by 50
  15. 2024Provably Robust Multi-bit Watermarking for AI-generated Textcited by 47
  16. 2024Pseudorandom Error-Correcting Codescited by 44
  17. 2024Copyright Protection in Generative AI: A Technical Perspectivecited by 38
  18. 2024PARDEN, Can You Repeat That? Defending against Jailbreaks via Repetitioncited by 35
  19. 2024Token-Specific Watermarking with Enhanced Detectability and Semantic Coherence for Large Language Modelscited by 34
  20. 2024On the Challenges and Opportunities in Generative AIcited by 31
  21. 2025Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Surveycited by 30
  22. 2023Embarrassingly Simple Text Watermarkscited by 29
  23. 2023Towards Possibilities & Impossibilities of AI-generated Text Detection: A Surveycited by 28
  24. 2023Towards Optimal Statistical Watermarkingcited by 26
  25. 2024Securing Large Language Models: Addressing Bias, Misinformation, and Prompt Attackscited by 26
  26. 2023Optimizing watermarks for large language modelscited by 25
  27. 2024COMPL-AI Framework: A Technical Interpretation and LLM Benchmarking Suite for the EU Artificial Intelligence Actcited by 24
  28. 2024Black-Box Detection of Language Model Watermarkscited by 23
  29. 2024Watermarking Techniques for Large Language Models: A Surveycited by 23
  30. 2024Learning to Watermark LLM-generated Text via Reinforcement Learningcited by 22
  31. 2024Theoretically Grounded Framework for LLM Watermarking: A Distribution-Adaptive Approachcited by 22
  32. 2024Double-I Watermark: Protecting Model Copyright for LLM Fine-tuningcited by 21
  33. 2023Necessary and Sufficient Watermark for Large Language Modelscited by 19
  34. 2024Debiasing Watermarks for Large Language Models via Maximal Couplingcited by 19
  35. 2024Watermarking Language Models for Many Adaptive Userscited by 18
  36. 2025Improving Your Model Ranking on Chatbot Arena by Vote Riggingcited by 17
  37. 2024A Watermark for Black-Box Language Modelscited by 16
  38. 2024Optimizing Adaptive Attacks against Watermarks for Language Modelscited by 16

50 most cited of 189 All 189 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Revisiting the Robustness of Watermarking to Paraphrasing Attacks
  2. 2023Mark My Words: Analyzing and Evaluating Language Model Watermarks
  3. 2024PostMark: A Robust Blackbox Watermark for Large Language Models
  4. 2023Provable Robust Watermarking for AI-Generated Text
  5. 2026AI Watermark Evidence Fails Forensic Readiness: An Empirical Evaluation
  6. 2023Robust Distortion-free Watermarks for Language Models
  7. 2024Provably Robust Watermarks for Open-Source Language Models
  8. 2025Signature vs. Substance: Evaluating the Balance of Adversarial Resistance and Linguistic Quality in Watermarking Large Language Models