mlsanityData to 7 Oct 2026

Valid Per-Field Selective Risk Control for Document Extraction: Three Failure Modes, a Validity Ladder, and When Conditioning Pays

arXiv 2608.14639 ↗First posted 28 Jul 2026cs.LG cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Per-field accept/review with selective risk at most alpha -- accept a field only if the error rate among accepted fields is controlled -- is the trust contract document-extraction systems need, and the natural procedure silently violates it on real documents. On 13,859 genuine claude-sonnet-5 fields from 800 CORD receipts (49.0% correct) we diagnose three failure modes: document clustering (design effect 1.84-2.45), score-refit leakage (coverage 0.416 at risk 0.127, violating alpha=0.10 in 95% of splits), and a tie-mass pathology (a degenerate score collapses the threshold grid, 0.030 to 0.001). We organize the fixes as a validity ladder, guarantee form stated per tier. A fit/val split protocol restores expected-selective-risk control for a learned fusion: coverage 0.318 at risk 0.096 at nominal alpha=0.10, no tolerance band (production variant 0.326) -- an on-average point whose realized risk exceeds alpha in 47.5% of resplits, not a certificate. Mondrian Learn-then-Test with exact binomial tails yields per-group PAC certificates: field-iid 0.171 at risk 0.068, cluster-corrected 0.140, doc-iid 0.060 -- the only tier matching documents, honestly near-vacuous today. Support-bin, the pre-specified provenance taxonomy, wins every rigor tier on the sonnet CORD capture (p<1e-4, Bonferroni-corrected) -- a win that does not replicate on the same documents under haiku or qwen -- while on higher-accuracy corpora pooled thresholds win: conditioning helps exactly where pooled cannot certify, subsumed by a learned score elsewhere. A frozen-configuration confirmation on selection-untouched claude-haiku-4-5 held at both risk levels, and a blind three-annotator human-gold audit verifies the practical tier's accepted-set risk at 1.3% against its 10% budget (Fleiss' kappa=0.83; labels err one-sidedly pessimistic). Released Apache-2.0 with seed-pinned, regression-gated procedures.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 16 of its 22 references that are in the map. First posted on arXiv 28 Jul 2026.

Builds on

16 of 22 references.

References · 16

  1. 2022Conformal Risk Controlcited by 290
  2. 2021Distribution-Free, Risk-Controlling Prediction Setscited by 278
  3. 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 256
  4. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 254
  5. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  6. 2023Conformal Prediction With Conditional Guaranteescited by 146
  7. 2024Language Models with Conformal Factuality Guaranteescited by 128
  8. 2023Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthinesscited by 121
  9. 2018Bias-Reduced Uncertainty Estimation for Deep Neural Classifierscited by 106
  10. 2023DocILE Benchmark for Document Information Localization and Extractioncited by 58
  11. 2024Overcoming Common Flaws in the Evaluation of Selective Classification Systemscited by 31
  12. 2024VISA: Retrieval Augmented Generation with Visual Source Attributioncited by 28
4 more
  1. 2025BoundingDocs: a Unified Dataset for Document Question Answering with Spatial Annotationscited by 8
  2. 2026ExtractBench: A Benchmark and Evaluation Methodology for Complex Structured Extractioncited by 8
  3. 2026Real-Time Trustworthiness Scoring for LLM Structured Outputs and Data Extractioncited by 3
  4. 2026Beyond Logprobs: A Multi-Signal Confidence Engine for LLM-Based Document Field Extractioncited by 2

Built on by

Not cited yet.