Grounded or Guessing? LVLM Confidence Estimation via Blind-Image Contrastive Ranking
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning
Abstract (arXiv)
Large vision-language models (LVLMs) suffer from visual ungroundedness: they can produce a fluent, confident, and even correct response driven entirely by language priors, with the image contributing nothing to the prediction. Existing confidence estimation methods cannot detect this, as they observe model behavior under normal inference with no mechanism to determine whether a prediction was shaped by the image or by text alone. We introduce BICR (Blind-Image Contrastive Ranking), a model-agnostic confidence estimation framework. The LVLM stays entirely frozen; instead, we extract its hidden states twice per sample, once with the real image-question pair and once with the image blacked out while the question is held fixed. A small, separate probe is then trained on the real-image hidden state, regularized by a ranking loss that penalizes higher confidence on the blacked-out view, teaching it to treat visual grounding as a signal of reliability. At inference only the single real-image pass is used, so BICR adds zero cost beyond a standard single-view probe. Evaluated across five modern LVLMs and seven baselines on a benchmark covering visual question answering, object hallucination detection, medical imaging, and financial document understanding, BICR achieves the best cross-LVLM average on both calibration and discrimination simultaneously, with statistically significant discrimination gains robust to cluster-aware analysis, at 4-18x fewer parameters than the strongest probing baseline.
Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. First posted on arXiv 11 May 2026.
Builds on
0 of 0 references.
No references here yet.
Built on by
Citing papers per year: 2026: 5.
5 papers.
- 2026Calibrated Triage, Not Autonomy: Confidence Estimation for Medical Vision-Language Modelscited by 2
- 2026Confidence Estimation for Financial Vision-Language Models in Chart and Document Understandingcited by 0
- 2026Uncertainty-Aware Decision Making in Multimodal Large Language Modelscited by 0
- 2026Separating Capability from Confidence: Grounded Dual-State Calibration for GRPO-Trained Medical Vision-Language Modelscited by 0
- 2026Evidence-Order Calibration for Selective Visual Reasoning under Progressive Loss of Question-Critical Evidencecited by 0
- 2026Separating Capability from Confidence: Grounded Dual-State Calibration for GRPO-Trained Medical Vision-Language Models6 Sep 2026
- 2026Evidence-Order Calibration for Selective Visual Reasoning under Progressive Loss of Question-Critical Evidence29 Aug 2026
- 2026Uncertainty-Aware Decision Making in Multimodal Large Language Models17 Aug 2026
- 2026Confidence Estimation for Financial Vision-Language Models in Chart and Document Understanding6 Aug 2026
- 2026Calibrated Triage, Not Autonomy: Confidence Estimation for Medical Vision-Language Models14 Jun 2026
Similar, not cited
Close in wording, no citation either way
- 2026LookBack: Where and How to Score LVLM Responses via Visual Reference Usage
- 2026VL-Calibration: Decoupled Confidence Calibration for Large Vision-Language Models Reasoning
- 2025To Trust Or Not To Trust Your Vision-Language Model's Prediction
- 2026Leveraging Visual Signals for Robust Token-Level Uncertainty in Vision-Language Generation
- 2026Scaling Test-Time Robustness of Vision-Language Models via Self-Critical Inference Framework
- 2026From Routing Signals to Selective Review: Visual regrounding in MoE VLMs
- 2026Small Vision-Language Models Know When They Are Wrong But Cannot Say So: A Two-Model Study of Stated versus Internal Confidence Under Realistic Image Degradation
- 2026The Mirage of Calibrated Confidence: Trajectory-Independence of Verbalized Confidence in Vision-Language Models