mlsanityData to 7 Oct 2026

Know What You do Not Know: Verbalized Uncertainty Estimation Robustness on Corrupted Images in Vision-Language Models

arXiv 2504.03440 ↗First posted 4 Apr 2025cs.CV cs.LGcited by 6

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

To leverage the full potential of Large Language Models (LLMs) it is crucial to have some information on their answers' uncertainty. This means that the model has to be able to quantify how certain it is in the correctness of a given response. Bad uncertainty estimates can lead to overconfident wrong answers undermining trust in these models. Quite a lot of research has been done on language models that work with text inputs and provide text outputs. Still, since the visual capabilities have been added to these models recently, there has not been much progress on the uncertainty of Visual Language Models (VLMs). We tested three state-of-the-art VLMs on corrupted image data. We found that the severity of the corruption negatively impacted the models' ability to estimate their uncertainty and the models also showed overconfidence in most of the experiments.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 25 references that are in the map. First posted on arXiv 4 Apr 2025.

Builds on

17 of 25 references.

References · 17

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017On Calibration of Modern Neural Networkscited by 4,860
  5. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  10. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  11. 2021Ethical and social risks of harm from Language Modelscited by 957
  12. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
5 more
  1. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  2. 2019Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Comingcited by 332
  3. 2024Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoningcited by 37
  4. 2015A Review Paper: Noise Models in Digital Image Processingcited by 23
  5. 2021I Find Your Lack of Uncertainty in Computer Vision Disturbingcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Modelscited by 16
  2. 2025Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missingcited by 14
  3. 2025Improving MLLM Historical Record Extraction with Test-Time Imagecited by 0
  4. 2026Small Vision-Language Models Know When They Are Wrong But Cannot Say So: A Two-Model Study of Stated versus Internal Confidence Under Realistic Image Degradationcited by 0
  5. 2026Bigger or Cheaper? Scale and Quantization Effects on Uncertainty Signals in Vision-Language Models Under Image Degradationcited by 0
  6. 2026Dissecting Training-Free Uncertainty Estimation in Multimodal Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Overconfidence is Key: Verbalized Uncertainty Evaluation in Large Language and Vision-Language Models
  2. 2025Are vision language models robust to uncertain inputs?
  3. 2024Uncertainty-Aware Evaluation for Vision-Language Models
  4. 2026Detecting Misbehaviors of Large Vision-Language Models by Evidential Uncertainty Quantification
  5. 2025The Role of Model Confidence on Bias Effects in Measured Uncertainties for Vision-Language Models
  6. 2025Diagnosing Corruption-Induced Reliability Failures in Vision-Language Models
  7. 2026Leveraging Visual Signals for Robust Token-Level Uncertainty in Vision-Language Generation
  8. 2026Uncertainty Is Not a Safety Net for Clinical VQA, but Can It Anticipate Model Failure?