mlsanityData to 7 Oct 2026

Who Gets Flagged? The Pluralistic Evaluation Gap in AI Content Watermarking

arXiv 2604.13776 ↗First posted 15 Apr 2026cs.CY cs.CL cs.CRcited by 2

On the map: LLM safety and security → Watermarking generated content and models

Abstract (arXiv)

Watermarking is becoming the default mechanism for AI content authentication, with governance policies and frameworks referencing it as infrastructure for content provenance. Yet across text, image, and audio modalities, watermark signal strength, detectability, and robustness depend on statistical properties of the content itself, properties that vary systematically across languages, cultural visual traditions, and demographic groups. We examine how this content dependence creates modality-specific pathways to bias. Reviewing the major watermarking benchmarks across modalities, we find that, with one exception, none report performance across languages, cultural content types, or population groups. To address this, we propose three concrete evaluation dimensions for pluralistic watermark benchmarking: cross-lingual detection parity, culturally diverse content coverage, and demographic disaggregation of detection metrics. We argue that watermarking is part of the pluralistic alignment pipeline and should be held to the same evaluation standards. We connect this to governance frameworks currently mandating watermarking deployment without requiring fairness evaluation. Our position is that evaluation must precede deployment, and that the same bias auditing requirements applied to AI models should extend to the verification layer.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Watermarking generated content and models, one of 12 directions in LLM safety and security. Builds on 35 of its 59 references that are in the map. First posted on arXiv 15 Apr 2026.

Builds on

35 of 59 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025Analyzing and Evaluating Unbiased Language Model Watermark28 Sep
  2. 2025Watermarking Without Standards Is Not AI Governance27 May
  3. 2025SoK: How Robust is Audio Watermarking in Generative AI models?24 Mar
  4. 2025Evaluating the Robustness and Accuracy of Text Watermarking Under Real-World Cross-Lingual Manipulations23 Feb
  5. 2024SoK: Watermarking for AI-Generated Content27 Nov
  6. 2024Watermark under Fire: A Robustness Evaluation of LLM Watermarking20 Nov
  7. 2024Robust Watermarking Using Generative Priors Against Image Editing: From Benchmarking to Advances24 Oct
  8. 2024PostMark: A Robust Blackbox Watermark for Large Language Models20 Jun
  9. 2024Gaussian Shading: Provable Performance-Lossless Image Watermarking for Diffusion Models7 Apr
  10. 2024Topic-Based Watermarks for Large Language Models2 Apr
  11. 2024Can Watermarks Survive Translation? On the Cross-lingual Consistency of Text Watermark for Large Language Models21 Feb
  12. 2023Mark My Words: Analyzing and Evaluating Language Model Watermarks1 Dec
3 more
  1. 2023WaterBench: Towards Holistic Evaluation of Watermarks for Large Language Models13 Nov
  2. 2023Tree-Ring Watermarks: Fingerprints for Diffusion Images that are Invisible and Robust31 May
  3. 2023A Watermark for Large Language Models24 Jan

Other references · 20

  1. 2022Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scalecited by 262
  2. 2019StegaStamp: Invisible Hyperlinks in Physical Photographscited by 256
  3. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 207
  4. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  5. 2024A Roadmap to Pluralistic Alignmentcited by 175
  6. 2019Does Object Recognition Work for Everyone?cited by 149
  7. 2024Benchmarking Vision Language Models for Cultural Understandingcited by 100
  8. 2018The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselinescited by 90
  9. 2024Proactive Detection of Voice Cloning with Localized Watermarkingcited by 67
  10. 2023WavMark: Watermarking for Audio Generationcited by 41
  11. 2023'Person' == Light-skinned, Western Man, and Sexualization of Women of Color: Stereotypes in Stable Diffusioncited by 33
  12. 2025SynthID-Image: Image watermarking at internet scalecited by 26
8 more
  1. 2024AudioMarkBench: Benchmarking Robustness of Audio Watermarkingcited by 19
  2. 2024Towards Geographic Inclusion in the Evaluation of Text-to-Image Modelscited by 16
  3. 2025A Comprehensive Real-World Assessment of Audio Watermarking Algorithms: Will They Survive Neural Codecs?cited by 10
  4. 2025"It's not a representation of me": Examining Accent Bias and Digital Exclusion in Synthetic AI Voice Servicescited by 9
  5. 2022Does lossy image compression affect racial bias within face recognition?cited by 5
  6. 2025VideoMarkBench: Benchmarking Robustness of Video Watermarkingcited by 4
  7. 2024Evaluating Durability: Benchmark Insights into Multimodal Watermarkingcited by 3
  8. 2024Sound Check: Auditing Audio Datasetscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.