# Object-Level Verbalized Confidence Calibration in Vision-Language Models via Semantic Perturbation

arXiv: https://arxiv.org/abs/2504.14848 · HTML: https://mlsanity.com/paper/2504.14848/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 18 of its 29 references that are in the map.
- First posted on arXiv 21 Apr 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Hallucination mitigation in LVLMs](https://mlsanity.com/mllm/hallucination-mitigation-in-lvlms/)

## Builds on (most cited first, 18 of 18)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning](https://mlsanity.com/paper/2408.05517/) (2024)
- [AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation](https://mlsanity.com/paper/2311.07397/) (2023)
- [A Survey of Confidence Estimation and Calibration in Large Language Models](https://mlsanity.com/paper/2311.08298/) (2023)
- [Conformal Prediction with Large Language Models for Multi-Choice Question Answering](https://mlsanity.com/paper/2305.18404/) (2023)
- [SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationales](https://mlsanity.com/paper/2405.20974/) (2024)
- [Confidence Under the Hood: An Investigation into the Confidence-Probability Alignment in Large Language Models](https://mlsanity.com/paper/2405.16282/) (2024)

## Built on by (most cited first, 12 of 12)

- [Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Models](https://mlsanity.com/paper/2505.20236/) (2025)
- [Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missing](https://mlsanity.com/paper/2507.10587/) (2025)
- [FESTA: Functionally Equivalent Sampling for Trust Assessment of Multimodal LLMs](https://mlsanity.com/paper/2509.16648/) (2025)
- [Grounded or Guessing? LVLM Confidence Estimation via Blind-Image Contrastive Ranking](https://mlsanity.com/paper/2605.10893/) (2026)
- [Instinct vs. Reflection: Unifying Token and Verbalized Confidence in Multimodal Large Models](https://mlsanity.com/paper/2604.17274/) (2026)
- [Just how sure are you? Improving Verbalized Uncertainty Calibration in Medical VQA](https://mlsanity.com/paper/2606.27023/) (2026)
- [MACD: Model-Aware Contrastive Decoding via Counterfactual Data](https://mlsanity.com/paper/2602.01740/) (2026)
- [UrbanAlign: Post-hoc Semantic Calibration for VLM-Human Preference Alignment](https://mlsanity.com/paper/2602.19442/) (2026)
- [Edge Reliability Gap in Vision-Language Models: Quantifying Failure Modes of Compressed VLMs Under Visual Corruption](https://mlsanity.com/paper/2603.26769/) (2026)
- [Separating Capability from Confidence: Grounded Dual-State Calibration for GRPO-Trained Medical Vision-Language Models](https://mlsanity.com/paper/2609.06419/) (2026)
- [Evidence-Order Calibration for Selective Visual Reasoning under Progressive Loss of Question-Critical Evidence](https://mlsanity.com/paper/2609.09184/) (2026)
- [Dissecting Training-Free Uncertainty Estimation in Multimodal Large Language Models](https://mlsanity.com/paper/2609.22206/) (2026)

## Abstract (arXiv)

Vision-language models (VLMs) excel in various multimodal tasks but frequently suffer from poor calibration, resulting in misalignment between their verbalized confidence and response correctness. This miscalibration undermines user trust, especially when models confidently provide incorrect or fabricated information. In this work, we propose a novel Confidence Calibration through Semantic Perturbation (CSP) framework to improve the calibration of verbalized confidence for VLMs in response to object-centric queries. We first introduce a perturbed dataset where Gaussian noise is applied to the key object regions to simulate visual uncertainty at different confidence levels, establishing an explicit mapping between visual ambiguity and confidence levels. We further enhance calibration through a two-stage training process combining supervised fine-tuning on the perturbed dataset with subsequent preference optimization. Extensive experiments on popular benchmarks demonstrate that our method significantly improves the alignment between verbalized confidence and response correctness while maintaining or enhancing overall task performance. These results highlight the potential of semantic perturbation as a practical tool for improving the reliability and interpretability of VLMs.