# LeakageBench: Document-Level Leakage Risk for Redacting Personally Identifiable Information in Document Images

arXiv: https://arxiv.org/abs/2609.02207 · HTML: https://mlsanity.com/paper/2609.02207/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 31 references that are in the map.
- First posted on arXiv 2 Sep 2026.

## Builds on (most cited first, 11 of 11)

- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019)
- [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://mlsanity.com/paper/2204.08387/) (2022)
- [FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents](https://mlsanity.com/paper/1905.13538/) (2019)
- [ICDAR2019 Competition on Scanned Receipt OCR and Information Extraction](https://mlsanity.com/paper/2103.10213/) (2021)
- [VRDU: A Benchmark for Visually-rich Document Understanding](https://mlsanity.com/paper/2211.15421/) (2022)
- [BuDDIE: A Business Document Dataset for Multi-task Information Extraction](https://mlsanity.com/paper/2404.04003/) (2024)
- [PRvL: Quantifying the Capabilities and Risks of Large Language Models for PII Redaction](https://mlsanity.com/paper/2508.05545/) (2025)
- [RedactBuster: Entity Type Recognition from Redacted Documents](https://mlsanity.com/paper/2404.12991/) (2024)
- [Is Your Model Sensitive? SPeDaC: A New Benchmark for Detecting and Classifying Sensitive Personal Data](https://mlsanity.com/paper/2208.06216/) (2022)

## Abstract (arXiv)

Real-world personally identifiable information (PII) redaction often operates on document images---scans, screenshots, and PDF renderings---where OCR errors, layout structure, and visual noise determine whether sensitive information is actually removed. Existing PII benchmarks are mostly text-centric and do not measure document-level redaction risk: a page remains unsafe if even one identifier is missed. We introduce LeakageBench, a challenge set of 500 document images with 11,954 GDPR-aligned PII annotations spanning direct identifiers, linkage keys, and contextual re-identification surfaces. We evaluate generic OCR pipelines, commercial and task-adapted OCR-dependent detectors, and OCR-free vision-language models using entity-level F1, group-wise leakage, and document-level leakage metrics. Code Interpreter raises GPT-5.5 localization F1 from 0.090 to 0.249, but critical page-level leakage remains 0.968. These results show that stronger detection and tool assistance improve localization without making most pages safe for release. LeakageBench provides a diagnostic benchmark for high-recall, spatially grounded PII redaction in document images.