# Beyond Logprobs: A Multi-Signal Confidence Engine for LLM-Based Document Field Extraction

arXiv: https://arxiv.org/abs/2606.24420 · HTML: https://mlsanity.com/paper/2606.24420/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 29 references that are in the map.
- First posted on arXiv 23 Jun 2026.

## Builds on (most cited first, 14 of 14)

- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [SelectiveNet: A Deep Neural Network with an Integrated Reject Option](https://mlsanity.com/paper/1901.09192/) (2019)
- [Addressing Failure Prediction by Learning Model Confidence](https://mlsanity.com/paper/1910.04851/) (2019)
- [Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthiness](https://mlsanity.com/paper/2308.16175/) (2023)
- [DocILE Benchmark for Document Information Localization and Extraction](https://mlsanity.com/paper/2302.05658/) (2023)
- [Self-Verification Improves Few-Shot Clinical Information Extraction](https://mlsanity.com/paper/2306.00024/) (2023)
- [LMDX: Language Model-based Document Information Extraction and Localization](https://mlsanity.com/paper/2309.10952/) (2023)
- [Real-Time Trustworthiness Scoring for LLM Structured Outputs and Data Extraction](https://mlsanity.com/paper/2603.18014/) (2026)
- [HYCEDIS: HYbrid Confidence Engine for Deep Document Intelligence System](https://mlsanity.com/paper/2206.02628/) (2022)
- [Confidence-Aware Document OCR Error Detection](https://mlsanity.com/paper/2409.04117/) (2024)

## Built on by (most cited first, 2 of 2)

- [Valid Per-Field Selective Risk Control for Document Extraction: Three Failure Modes, a Validity Ladder, and When Conditioning Pays](https://mlsanity.com/paper/2608.14639/) (2026)
- [Perception, Layout, and Validation: Calibrated Confidence for Reliable Straight-Through Processing of Financial Documents](https://mlsanity.com/paper/2609.20110/) (2026)

## Abstract (arXiv)

In high-stakes document processing pipelines, including financial reconciliation, compliance verification, and procurement automation, an LLM extraction that is silently wrong is more dangerous than one that is visibly absent. The central challenge is not extraction accuracy alone but reliable confidence estimation: knowing, field by field, whether an extraction can be trusted for automation or deferred to human review. Token-level log-probabilities, verbalized confidence, and multi-sample self-consistency all collapse toward all-positive behaviour at practical thresholds, offering no reliable separation between trustworthy and untrustworthy extractions. We present ExtractConf, a cross-domain, field-agnostic confidence engine that grounds confidence estimation in two structurally different readings of the same document. A field-guided Hunter call extracts each field under schema-slot completion pressure; a document-guided Mapper call scans holistically and surfaces values grounded in document content. This asymmetry yields different failure modes: Hunter hallucinates values for absent fields, while Mapper misses visually non-salient ones. Their disagreement is independently informative. ExtractConf fuses cross-call disagreement, LLM-internal uncertainty, OCR, image quality, and spatial layout into a classifier requiring no domain-specific rules or retraining. On DocILE (55-field invoices, 26% failure rate), it achieves 0.928 ROC AUC and reduces selective prediction risk by 70% over logprob-mean. At 80% coverage, accuracy reaches 99.1%, enabling a practical human-in-the-loop workflow. Zero-shot transfer to CORD receipts achieves 0.858 AUC; lightweight Lasso recalibration reduces ECE by 89% and Brier by 43%, confirming the signals generalise across document domains.