# Disappearing Ink: Obfuscation Breaks N-gram Code Watermarks in Theory and Practice

arXiv: https://arxiv.org/abs/2507.05512 · HTML: https://mlsanity.com/paper/2507.05512/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 73 references that are in the map.
- First posted on arXiv 7 Jul 2025.

## Builds on (most cited first, 29 of 29)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021)
- [Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense](https://mlsanity.com/paper/2303.13408/) (2023)
- [Can AI-Generated Text be Reliably Detected?](https://mlsanity.com/paper/2303.11156/) (2023)
- [CodeT: Code Generation with Generated Tests](https://mlsanity.com/paper/2207.10397/) (2022)
- [Robust Distortion-free Watermarks for Language Models](https://mlsanity.com/paper/2307.15593/) (2023)
- [CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X](https://mlsanity.com/paper/2303.17568/) (2023)
- [Provable Robust Watermarking for AI-Generated Text](https://mlsanity.com/paper/2306.17439/) (2023)
- [Generating Natural Adversarial Examples](https://mlsanity.com/paper/1710.11342/) (2017)
- [MapCoder: Multi-Agent Code Generation for Competitive Problem Solving](https://mlsanity.com/paper/2405.11403/) (2024)
- [Software Testing with Large Language Models: Survey, Landscape, and Vision](https://mlsanity.com/paper/2307.07221/) (2023)
- [Who Wrote this Code? Watermarking for Code Generation](https://mlsanity.com/paper/2305.15060/) (2023)
- [Robust Neural Machine Translation with Doubly Adversarial Inputs](https://mlsanity.com/paper/1906.02443/) (2019)
- [Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples](https://mlsanity.com/paper/1803.01128/) (2018)
- [On Adversarial Examples for Character-Level Neural Machine Translation](https://mlsanity.com/paper/1806.09030/) (2018)
- [RankGen: Improving Text Generation with Large Ranking Models](https://mlsanity.com/paper/2205.09726/) (2022)
- [An Unforgeable Publicly Verifiable Watermark for Large Language Models](https://mlsanity.com/paper/2307.16230/) (2023)
- [GumbelSoft: Diversified Language Model Watermarking via the GumbelMax-trick](https://mlsanity.com/paper/2402.12948/) (2024)
- [CodeMark: Imperceptible Watermarking for Code Datasets against Neural Code Completion Models](https://mlsanity.com/paper/2308.14401/) (2023)
- [CodeIP: A Grammar-Guided Multi-Bit Watermark for Large Language Models of Code](https://mlsanity.com/paper/2404.15639/) (2024)
- [Permute-and-Flip: An optimally stable and watermarkable decoder for LLMs](https://mlsanity.com/paper/2402.05864/) (2024)
- [Rapidly Mixing Markov Chains: A Comparison of Techniques (A Survey)](https://mlsanity.com/paper/1603.01512/) (2016)

## Abstract (arXiv)

Large language models (LLMs) are increasingly used for code generation, making reliable identification of machine-generated code important for attribution, tracking, and misuse detection. Existing code watermarking methods are dominated by N-gram-based schemes, yet their robustness has mostly been evaluated only against simple edits or optimizations. We argue that this significantly overstates security, because software engineering already provides stronger semantics-preserving transformations in the form of code obfuscation. We study N-gram-based code watermarking under obfuscation. We formally model semantics-preserving transformations as a Markov random walk and prove that, under an intuitive and experimentally supported assumption called distribution consistency, obfuscation can nullify the robustness of N-gram-based watermarks. If the original detector has a false positive rate fpr, then after obfuscation, its failure rate on watermarked code approaches 1 - fpr. We validate this theory on three state-of-the-art watermarking schemes, two LLMs, two programming languages, four benchmarks, and four obfuscators. Across all settings, detectors collapse to near-random performance on obfuscated code (AUROC tightly around 0.5), and for each language, at least one attack leaves all post-obfuscation AUROC scores below 0.6. These results jointly show that current N-gram-based code watermarks are not robust to realistic obfuscation attacks and motivate more semantics-aware alternatives.