mlsanityData to 14 Oct 2026

LeakageBench: Document-Level Leakage Risk for Redacting Personally Identifiable Information in Document Images

arXiv 2609.02207 ↗First posted 2 Sep 2026cs.CV cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Real-world personally identifiable information (PII) redaction often operates on document images---scans, screenshots, and PDF renderings---where OCR errors, layout structure, and visual noise determine whether sensitive information is actually removed. Existing PII benchmarks are mostly text-centric and do not measure document-level redaction risk: a page remains unsafe if even one identifier is missed. We introduce LeakageBench, a challenge set of 500 document images with 11,954 GDPR-aligned PII annotations spanning direct identifiers, linkage keys, and contextual re-identification surfaces. We evaluate generic OCR pipelines, commercial and task-adapted OCR-dependent detectors, and OCR-free vision-language models using entity-level F1, group-wise leakage, and document-level leakage metrics. Code Interpreter raises GPT-5.5 localization F1 from 0.090 to 0.249, but critical page-level leakage remains 0.968. These results show that stronger detection and tool assistance improve localization without making most pages safe for release. LeakageBench provides a diagnostic benchmark for high-recall, spatially grounded PII redaction in document images.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 11 of its 31 references that are in the map. First posted on arXiv 2 Sep 2026.

Builds on

11 of 31 references.

References · 11

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  2. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  3. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 553
  4. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 485
  5. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 349
  6. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 256
  7. 2022VRDU: A Benchmark for Visually-rich Document Understandingcited by 35
  8. 2024BuDDIE: A Business Document Dataset for Multi-task Information Extractioncited by 8
  9. 2025PRvL: Quantifying the Capabilities and Risks of Large Language Models for PII Redactioncited by 7
  10. 2024RedactBuster: Entity Type Recognition from Redacted Documentscited by 4
  11. 2022Is Your Model Sensitive? SPeDaC: A New Benchmark for Detecting and Classifying Sensitive Personal Datacited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Visual Evidence: Revealing and Mitigating Relational Privacy Leakage in Document MLLMs
  2. 2026PII-VisBench: Evaluating Personally Identifiable Information Safety in Vision Language Models Along a Continuum of Visibility
  3. 2026RedactionBench
  4. 2026WebPII: Benchmarking Visual PII Detection for Computer-Use Agents
  5. 2026Assessing Privacy Preservation and Utility in Online Vision-Language Models
  6. 2025Vision Token Masking Alone Cannot Prevent PHI Leakage in Medical Document OCR: A Systematic Evaluation
  7. 2026VisualLeakBench: Auditing the Fragility of Large Vision-Language Models against PII Leakage and Social Engineering
  8. 2025On the Robustness of Vision-Language Models in Zero-shot Privacy Classification