mlsanityData to 7 Oct 2026

Can You Trust the Confidence? ConfBench for Vision-Language Models on Document Extraction

arXiv 2608.01792 ↗First posted 3 Aug 2026cs.AI cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Intelligent document processing (IDP) with vision-language models (VLMs) hinges on confidence scores trustworthy enough to route extractions between automation and human review. Existing document benchmarks are dominated by clean, high-quality samples, leaving low accuracy regions too sparse for calibration assessment. We introduce ConfBench, the first calibration-specific benchmark for key information extraction (KIE), built by applying 20 controlled degradation pipelines to a diverse document set, yielding 1,346 variants and 70K+ entity-level evaluations spanning the full accuracy spectrum. We evaluate four proprietary and three open-weight VLMs under verbalized and log-probability confidence estimation methods across three input modalities, and find: (i) OCR+Image modality results in more accurate confidence estimates; (ii) model capability is the dominant factor: within the Claude family confidence quality scales monotonically with capability, while across families parameter count is a poor predictor; (iii) calibration quality varies widely across models, from near-perfect to severely overconfident, and per-model post-hoc correction rescales these absolute confidence values for threshold-based routing without altering ranking-based operational metrics; and (iv) log-probability with first-token aggregation consistently outperforms mean-token and margin aggregations. We also introduce ECARB, a review-budget metric translating discriminative gains into operational savings. We release ConfBench publicly to enable systematic study of confidence estimators and calibration methods for trustworthy IDP application deployment.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 14 of its 29 references that are in the map. First posted on arXiv 3 Aug 2026.

Builds on

14 of 29 references.

Closest ancestors · 1

Same direction, newest first

  1. 2026UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual Documents3 Feb

Other references · 13

  1. 2017On Calibration of Modern Neural Networkscited by 4,860
  2. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  3. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  4. 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 212
  5. 2023DocILE Benchmark for Document Information Localization and Extractioncited by 58
  6. 2024Calibrating Language Models with Adaptive Temperature Scalingcited by 57
  7. 2022Sample-dependent Adaptive Temperature Scaling for Improved Calibrationcited by 48
  8. 2024Think Twice Before Trusting: Self-Detection for Large Language Models through Comprehensive Answer Reflectioncited by 40
  9. 2024Are Large Language Models More Honest in Their Probabilistic or Verbalized Confidence?cited by 23
  10. 2024Human-in-the-loop or AI-in-the-loop? Automate or Collaborate?cited by 23
  11. 2024A Survey of Calibration Process for Black-Box LLMscited by 8
  12. 2026DocSplit: A Comprehensive Benchmark Dataset and Evaluation Approach for Document Packet Recognition and Splittingcited by 5
1 more
  1. 2024RealKIE: Five Novel Datasets for Enterprise Key Information Extractioncited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Accuracy: Robustness, Cost, and Governance Trade-offs for Vision-Language Models in Templated Document Extraction
  2. 2026Beyond Logprobs: A Multi-Signal Confidence Engine for LLM-Based Document Field Extraction
  3. 2024Patchfinder: Leveraging Visual Language Models for Accurate Information Retrieval using Model Uncertainty
  4. 2026Perception, Layout, and Validation: Calibrated Confidence for Reliable Straight-Through Processing of Financial Documents
  5. 2026RaV-IDP: A Reconstruction-as-Validation Framework for Faithful Intelligent Document Processing
  6. 2026Evaluating Structured Information Extraction with Open Models in a High Risk Public Sector Application
  7. 2025Trust the Model: Compact VLMs as In-Context Judges for Image-Text Data Quality
  8. 2026Real5-OmniDocBench: A Full-Scale Physical Reconstruction Benchmark for Robust Document Parsing in the Wild