mlsanityData to 7 Oct 2026

Taming Object Hallucinations with Verified Atomic Confidence Estimation

arXiv 2511.09228 ↗First posted 12 Nov 2025cs.CV cs.CLcited by 4

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) often suffer from hallucinations, particularly errors in object existence, attributes, or relations, which undermine their reliability. We introduce TACO (Verified Atomic Confidence Estimation), a simple framework that mitigates hallucinations through self-verification and confidence calibration without relying on external vision experts. TACO decomposes responses into atomic queries, paraphrases them to reduce sensitivity to wording, and estimates confidence using self-consistency (black-box) or self-confidence (gray-box) aggregation, before refining answers with a language model. Experiments on five benchmarks (POPE, MME, HallusionBench, AMBER, and MM-Hal Bench) with two MLLMs (textttLLaVA-1.5-7B and textttCogVLM2) show that TACO consistently outperforms direct prompting and Visual Contrastive Decoding, reduces systematic biases, and improves confidence calibration, demonstrating its effectiveness in enhancing the faithfulness of MLLMs.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 51 references that are in the map. First posted on arXiv 12 Nov 2025.

Builds on

35 of 51 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Pelican: Correcting Hallucination in Vision-LLMs via Claim Decomposition and Program of Thought Verification2 Jul
  2. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  3. 2024Multi-Modal Hallucination Control by Visual Information Grounding20 Mar
  4. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  5. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  6. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data22 Nov
  7. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov

Other references · 28

  1. 2017On Calibration of Modern Neural Networkscited by 4,860
  2. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  4. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  5. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  6. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  7. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  8. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  9. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  10. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  11. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  12. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
16 more
  1. 2021A Survey on Automated Fact-Checkingcited by 376
  2. 2020Calibration of Pre-trained Transformerscited by 307
  3. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 304
  4. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  5. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  6. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 209
  7. 2020Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learningcited by 190
  8. 2023Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Modelscited by 91
  9. 2024Calibrating Large Language Models with Sample Consistencycited by 72
  10. 2019Reasoning Over Semantic-Level Graph for Fact Checkingcited by 69
  11. 2020On the Inference Calibration of Neural Machine Translationcited by 61
  12. 2024Linguistic Calibration of Long-Form Generationscited by 59
  13. 2022Detection of Word Adversarial Examples in Text Classification: Benchmark and Baseline via Robust Density Estimationcited by 32
  14. 2024Large Language Model Confidence Estimation via Black-Box Accesscited by 14
  15. 2025Promptception: How Sensitive Are Large Multimodal Models to Prompts?cited by 13
  16. 2023Great Models Think Alike: Improving Model Reliability via Inter-Model Latent Agreementcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026The Confidence Dichotomy: Analyzing and Mitigating Miscalibration in Tool-Use Agentscited by 14
  2. 2026Re-Centering Humans in LLM Personalizationcited by 0
  3. 2026Witness Evidence Portfolios: Single-Prefill Risk Detection for Closed Multimodal Answerscited by 0
  4. 2026RECAP: Relation Evidence Calibration for Detecting Spatial Relation Hallucinations in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023FaithScore: Fine-grained Evaluations of Hallucinations in Large Vision-Language Models
  2. 2025Preemptive Hallucination Reduction: An Input-Level Approach for Multimodal Language Model
  3. 2025Self-Consistency as a Free Lunch: Reducing Hallucinations in Vision-Language Models via Self-Reflection
  4. 2025Mitigating Multimodal Hallucinations via Gradient-based Self-Reflection
  5. 2025The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs?
  6. 2025Mitigating Hallucinations in Large Vision-Language Models with Internal Fact-based Contrastive Decoding
  7. 2024ODE: Open-Set Evaluation of Hallucinations in Multimodal Large Language Models
  8. 2026Rethinking Visual Neglect: Steering via Context-Preference for MLLM Hallucination Mitigation