mlsanityData to 7 Oct 2026

Watermark Stealing in Large Language Models

arXiv 2402.19361 ↗First posted 29 Feb 2024cs.LG cs.AI cs.CRcited by 79

On the map: LLM safety and security → Watermarking generated content and models

Abstract (arXiv)

LLM watermarking has attracted attention as a promising way to detect AI-generated content, with some works suggesting that current schemes may already be fit for deployment. In this work we dispute this claim, identifying watermark stealing (WS) as a fundamental vulnerability of these schemes. We show that querying the API of the watermarked LLM to approximately reverse-engineer a watermark enables practical spoofing attacks, as hypothesized in prior work, but also greatly boosts scrubbing attacks, which was previously unnoticed. We are the first to propose an automated WS algorithm and use it in the first comprehensive study of spoofing and scrubbing in realistic settings. We show that for under 50 an attacker can both spoof and scrub state-of-the-art schemes previously considered safe, with average success rate of over 80%. Our findings challenge common beliefs about LLM watermarking, stressing the need for more robust schemes. We make all our code and additional examples available at https://watermark-stealing.org.

Key facts. Cited by 79 papers in the map (data through 7 Oct 2026). Placed in Watermarking generated content and models, one of 12 directions in LLM safety and security. Builds on 36 of its 54 references that are in the map. First posted on arXiv 29 Feb 2024.

Builds on

36 of 54 references.

Closest ancestors · 18

Same direction, newest first

  1. 2023A Survey of Text Watermarking in the Era of Large Language Models13 Dec
  2. 2023On the Learnability of Watermarks for Language Models7 Dec
  3. 2023Mark My Words: Analyzing and Evaluating Language Model Watermarks1 Dec
  4. 2023A Robust Semantics-based Watermark for Large Language Model against Paraphrasing15 Nov
  5. 2023A Semantic Invariant Robust Watermark for Large Language Models10 Oct
  6. 2023Necessary and Sufficient Watermark for Large Language Models2 Oct
  7. 2023Unbiased Watermark for Large Language Models22 Sep
  8. 2023Robust Distortion-free Watermarks for Language Models28 Jul
  9. 2023Provable Robust Watermarking for AI-Generated Text30 Jun
  10. 2023On the Reliability of Watermarks for Large Language Models7 Jun
  11. 2023Are You Copying My Model? Protecting the Copyright of Large Language Models for EaaS via Backdoor Watermark17 May
  12. 2023Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense23 Mar
6 more
  1. 2023Can AI-Generated Text be Reliably Detected?17 Mar
  2. 2023Protecting Language Generation Models via Invisible Watermarking6 Feb
  3. 2023A Watermark for Large Language Models24 Jan
  4. 2022Distillation-Resistant Watermarking for Model Protection in NLP7 Oct
  5. 2022CATER: Intellectual Property Protection on Text Generation APIs via Conditional Watermarks19 Sep
  6. 2021Protecting Intellectual Property of Language Generation APIs with Lexical Watermark5 Dec

Other references · 18

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  7. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  8. 2021Membership Inference Attacks From First Principlescited by 839
  9. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  10. 2023DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvaturecited by 705
  11. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  12. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 191
6 more
  1. 2023RADAR: Robust AI-Text Detection via Adversarial Learningcited by 187
  2. 2020Adversarial Watermarking Transformer: Towards Tracing Text Provenance with Data Hidingcited by 121
  3. 2019Neural Linguistic Steganographycited by 73
  4. 2023Towards Possibilities & Impossibilities of AI-generated Text Detection: A Surveycited by 28
  5. 2021Paraphrastic Representations at Scalecited by 19
  6. 2023On the Zero-Shot Generalization of Machine-Generated Text Detectorscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 19; 2025: 35; 2026: 25.

79 papers.

Sort:

By area:

  1. 2024SoK: Watermarking for AI-Generated Contentcited by 43
  2. 2025SynthID-Image: Image watermarking at internet scalecited by 26
  3. 2024How to Trace Latent Generative Model Generated Images without Artificial Watermark?cited by 25
  4. 2024COMPL-AI Framework: A Technical Interpretation and LLM Benchmarking Suite for the EU Artificial Intelligence Actcited by 23
  5. 2024Black-Box Detection of Language Model Watermarkscited by 22
  6. 2024Bileve: Securing Text Provenance in Large Language Models Against Spoofing with Bi-level Signaturecited by 21
  7. 2026Large Language Model Reasoning Failurescited by 20
  8. 2024Revisiting the Robustness of Watermarking to Paraphrasing Attackscited by 19
  9. 2025BiMark: Unbiased Multilayer Watermarking for Large Language Modelscited by 18
  10. 2024Self-Recognition in Language Modelscited by 17
  11. 2025Adversarial Paraphrasing: A Universal Attack for Humanizing AI-Generated Textcited by 16
  12. 2024Ward: Provable RAG Dataset Inference via LLM Watermarkscited by 15
38 more
  1. 2025Watermarking Autoregressive Image Generationcited by 15
  2. 2025Mark Your LLM: Detecting the Misuse of Open-Source Large Language Models via Watermarkingcited by 14
  3. 2025Revealing Weaknesses in Text Watermarking Through Self-Information Rewrite Attackscited by 12
  4. 2024Optimizing Adaptive Attacks against Watermarks for Language Modelscited by 11
  5. 2024De-mark: Watermark Removal in Large Language Modelscited by 11
  6. 2025Towards Watermarking of Open-Source LLMscited by 11
  7. 2025AI Safety Should Prioritize the Future of Workcited by 11
  8. 2025StealthInk: A Multi-bit and Stealthy Watermark for Large Language Modelscited by 11
  9. 2024Watermarking Language Models with Error Correcting Codescited by 10
  10. 2024Can Watermarked LLMs be Identified by Users via Crafted Prompts?cited by 10
  11. 2025Unified attacks to large language model watermarks: spoofing and scrubbing in unauthorized knowledge distillationcited by 9
  12. 2025Watermarking Diffusion Language Modelscited by 9
  13. 2025Enhancing LLM Watermark Resilience Against Both Scrubbing and Spoofing Attackscited by 8
  14. 2024Topic-Based Watermarks for Large Language Modelscited by 7
  15. 2025LLM Fingerprinting via Semantically Conditioned Watermarkscited by 7
  16. 2025Character-Level Perturbations Disrupt LLM Watermarkscited by 7
  17. 2025Are Robust LLM Fingerprints Adversarially Robust?cited by 7
  18. 2025Position: LLM Watermarking Should Align Stakeholders' Incentives for Practical Adoptioncited by 7
  19. 2024ESPERANTO: Evaluating Synthesized Phrases to Enhance Robustness in AI Detection for Text Originationcited by 6
  20. 2024Humanizing the Machine: Proxy Attacks to Mislead LLM Detectorscited by 6
  21. 2024Survey on Plagiarism Detection in Large Language Models: The Impact of ChatGPT and Gemini on Academic Integritycited by 5
  22. 2024B4: A Black-Box Scrubbing Attack on LLM Watermarkscited by 5
  23. 2025From Trade-off to Synergy: A Versatile Symbiotic Watermarking Framework for Large Language Modelscited by 5
  24. 2025BitMark: Watermarking Bitwise Autoregressive Image Generative Modelscited by 5
  25. 2025On the Empirical Power of Goodness-of-Fit Tests in Watermark Detectioncited by 5
  26. 2025Watermarking Discrete Diffusion Language Modelscited by 5
  27. 2024Discovering Spoofing Attempts on Language Model Watermarkscited by 4
  28. 2025Can you Finetune your Binoculars? Embedding Text Watermarks into the Weights of Large Language Modelscited by 4
  29. 2025Mitigating Watermark Forgery in Generative Models via Randomized Key Selectioncited by 4
  30. 2025DMark: Order-Agnostic Watermarking for Diffusion Large Language Modelscited by 4
  31. 2025Majority Bit-Aware Watermarking For Large Language Modelscited by 3
  32. 2025On the Detectability of LLM-Generated Text: What Exactly Is LLM-Generated Text?cited by 3
  33. 2026Sequential Behavioral Watermarking for LLM Agentscited by 3
  34. 2026TRACE: A Two-Channel Robust Attribution Watermark via Complementary Embeddings for LLM-Agent Trajectoriescited by 3
  35. 2025SoK: Are Watermarks in LLMs Ready for Deployment?cited by 2
  36. 2025SimKey: A Semantically Aware Key Module for Watermarking Language Modelscited by 2
  37. 2025Verifiable Fine-Tuning for LLMs: Zero-Knowledge Training Proofs Bound to Data Provenance and Policycited by 2
  38. 2025Advancing Machine-Generated Text Detection from an Easy to Hard Supervision Perspectivecited by 2

50 most cited of 79 All 79 (JSON)