mlsanityData to 7 Oct 2026

Invisible Ink, Visible Lies: How Production Watermarking Causes LLMs to Hallucinate

arXiv 2610.04860 ↗First posted 4 Oct 2026cs.CR cs.AIcited by 0

On the map: LLM safety and security → Watermarking generated content and models

Abstract (arXiv)

Text watermarking helps identify AI-generated content, but its effect on factual reliability remains underexplored. In this paper, we study watermarking hallucination: factual errors induced or amplified by watermarking even when the required evidence is present in the context and the unwatermarked model can answer correctly. Using a controlled retrieval-augmented generation setting, we compare unwatermarked and watermarked generations under the same context, query, and decoding configuration, and quantify their factual accuracy decrease. Across six representative watermarking methods, including KGW, SWEET, DiPmark, GumbelSoft, Gumbel-Max, and SynthID watermarking, we consistently observe watermark-induced hallucination. Watermarked outputs can remain fluent while introducing factual errors. We attribute this failure mode to two mechanisms: (1) token perturbations in the current-step arising from method-specific reweighting or keyed sampling, and (2) prefix-induced attention drift, which accumulates through autoregressive decoding and weakens later attention to the factual context. Motivated by this analysis, we propose two plug-in interventions at the token and attention levels that can be integrated into existing watermarking methods to improve factuality. At a matched TPR of 0.90 at 1% FPR, combining the two interventions reduces factual errors by approximately 90% relative to watermark-only decoding while preserving fluency and comparable decoding efficiency. Overall, this work highlights factuality as a first-class criterion in watermark evaluation, alongside detectability and robustness, and calls for careful factuality validation before deploying watermarks in fact-critical applications.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Watermarking generated content and models, one of 12 directions in LLM safety and security. First posted on arXiv 4 Oct 2026.

Builds on

No reference list yet.

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Addressing Image Hallucination in Text-to-Image Generation through Factual Image Retrieval
  2. 2026From Architecture to Output: Structural Origins of Hallucination in Large Language Models and the Amplifying Role of Data
  3. 2026Improving the Trade-off Between Watermark Strength and Speculative Sampling Efficiency for Language Models
  4. 2025Beyond Accuracy: Rethinking Hallucination and Regulatory Response in Generative AI
  5. 2024AIGCs Confuse AI Too: Investigating and Explaining Synthetic Image-induced Hallucinations in Large Vision-Language Models
  6. 2023WatME: Towards Lossless Watermarking Through Lexical Redundancy
  7. 2026Rethinking Forgery Attacks on Semantic Watermarks in Black-Box Settings: A Geometric Distortion Perspective
  8. 2025The Curious Case of Factuality Finetuning: Models' Internal Beliefs Can Improve Factuality