# De-mark: Watermark Removal in Large Language Models

arXiv: https://arxiv.org/abs/2410.13808 · HTML: https://mlsanity.com/paper/2410.13808/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Watermarking generated content and models, one of 12 directions in LLM safety and security.
- Builds on 32 of its 52 references that are in the map.
- First posted on arXiv 17 Oct 2024.

## Placed in

- LLM safety and security / [Watermarking generated content and models](https://mlsanity.com/safety/watermarking-generated-content-and-models/)

## Builds on (most cited first, 30 of 32)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [A Watermark for Large Language Models](https://mlsanity.com/paper/2301.10226/) (2023)
- [VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks](https://mlsanity.com/paper/2305.11175/) (2023)
- [AudioPaLM: A Large Language Model That Can Speak and Listen](https://mlsanity.com/paper/2306.12925/) (2023)
- [Robust Distortion-free Watermarks for Language Models](https://mlsanity.com/paper/2307.15593/) (2023)
- [Provable Robust Watermarking for AI-Generated Text](https://mlsanity.com/paper/2306.17439/) (2023)
- [Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions](https://mlsanity.com/paper/2308.11483/) (2023)
- [Undetectable Watermarks for Language Models](https://mlsanity.com/paper/2306.09194/) (2023)
- [On the Reliability of Watermarks for Large Language Models](https://mlsanity.com/paper/2306.04634/) (2023)
- [A Semantic Invariant Robust Watermark for Large Language Models](https://mlsanity.com/paper/2310.06356/) (2023)
- [Unbiased Watermark for Large Language Models](https://mlsanity.com/paper/2310.10669/) (2023)
- [Watermark Stealing in Large Language Models](https://mlsanity.com/paper/2402.19361/) (2024)
- [Mark My Words: Analyzing and Evaluating Language Model Watermarks](https://mlsanity.com/paper/2312.00273/) (2023)
- [Towards Codable Watermarking for Injecting Multi-bits Information to LLMs](https://mlsanity.com/paper/2307.15992/) (2023)
- [Image Watermarks are Removable Using Controllable Regeneration from Clean Noise](https://mlsanity.com/paper/2410.05470/) (2024)
- [Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selection](https://mlsanity.com/paper/2402.12501/) (2024)
- [Token-Specific Watermarking with Enhanced Detectability and Semantic Coherence for Large Language Models](https://mlsanity.com/paper/2402.18059/) (2024)
- [Split-and-Denoise: Protect large language model inference with local differential privacy](https://mlsanity.com/paper/2310.09130/) (2023)
- [Black-Box Detection of Language Model Watermarks](https://mlsanity.com/paper/2405.20777/) (2024)
- [Adversarial Weight Perturbation Improves Generalization in Graph Neural Networks](https://mlsanity.com/paper/2212.04983/) (2022)
- [Learning to Learn from APIs: Black-Box Data-Free Meta-Learning](https://mlsanity.com/paper/2305.18413/) (2023)
- [RetrievalGuard: Provably Robust 1-Nearest Neighbor Image Retrieval](https://mlsanity.com/paper/2206.11225/) (2022)
- [Few-Shot Class Incremental Learning with Attention-Aware Self-Adaptive Prompt](https://mlsanity.com/paper/2403.09857/) (2024)
- [Large Language Model Watermark Stealing With Mixed Integer Programming](https://mlsanity.com/paper/2405.19677/) (2024)
- [Can Watermarked LLMs be Identified by Users via Crafted Prompts?](https://mlsanity.com/paper/2410.03168/) (2024)
- [Distortion-free Watermarks are not Truly Distortion-free under Watermark Key Collisions](https://mlsanity.com/paper/2406.02603/) (2024)

## Built on by (most cited first, 14 of 14)

- [Character-Level Perturbations Disrupt LLM Watermarks](https://mlsanity.com/paper/2509.09112/) (2025)
- [An Ensemble Framework for Unbiased Language Model Watermarking](https://mlsanity.com/paper/2509.24043/) (2025)
- [Analyzing and Evaluating Unbiased Language Model Watermark](https://mlsanity.com/paper/2509.24048/) (2025)
- [DMark: Order-Agnostic Watermarking for Diffusion Large Language Models](https://mlsanity.com/paper/2510.02902/) (2025)
- [A Visual Semantic Adaptive Watermark grounded by Prefix-Tuning for Large Vision-Language Model](https://mlsanity.com/paper/2601.07291/) (2026)
- [Towards Robust Content Watermarking Against Removal and Forgery Attacks](https://mlsanity.com/paper/2604.06662/) (2026)
- [RLSpoofer: A Lightweight Evaluator for LLM Watermark Spoofing Resilience](https://mlsanity.com/paper/2604.11546/) (2026)
- [SIF: Semantically In-Distribution Fingerprints for Large Vision-Language Models](https://mlsanity.com/paper/2604.17041/) (2026)
- [LLM Watermark Evasion via Bias Inversion](https://mlsanity.com/paper/2509.23019/) (2025)
- [Semantic Watermarking with Order-Robust Detection over Sub-sentence Units](https://mlsanity.com/paper/2608.27666/) (2026)
- [Breaking Distortion-free Watermarks in Large Language Models](https://mlsanity.com/paper/2502.18608/) (2025)
- [GoCoMA: Hyperbolic Multimodal Representation Fusion for Large Language Model-Generated Code Attribution](https://mlsanity.com/paper/2604.16377/) (2026)
- [A Regulatory Placebo? The Systemic Failure of Mandatory GenAI Labeling](https://mlsanity.com/paper/2608.16470/) (2026)
- [TANGO: Watermarking Masked Diffusion Language Models in Token Pairs](https://mlsanity.com/paper/2609.35224/) (2026)

## Abstract (arXiv)

Watermarking techniques offer a promising way to identify machine-generated content via embedding covert information into the contents generated from language models (LMs). However, the robustness of the watermarking schemes has not been well explored. In this paper, we present De-mark, an advanced framework designed to remove n-gram-based watermarks effectively. Our method utilizes a novel querying strategy, termed random selection probing, which aids in assessing the strength of the watermark and identifying the red-green list within the n-gram watermark. Experiments on popular LMs, such as Llama3 and ChatGPT, demonstrate the efficiency and effectiveness of De-mark in watermark removal and exploitation tasks.