mlsanityData to 7 Oct 2026

Decompose and Compare Consistency: Measuring VLMs' Answer Reliability via Task-Decomposition Consistency Comparison

arXiv 2407.07840 ↗First posted 10 Jul 2024cs.CV cs.CLcited by 7

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Despite tremendous advancements, current state-of-the-art Vision-Language Models (VLMs) are still far from perfect. They tend to hallucinate and may generate biased responses. In such circumstances, having a way to assess the reliability of a given response generated by a VLM is quite useful. Existing methods, such as estimating uncertainty using answer likelihoods or prompt-based confidence generation, often suffer from overconfidence. Other methods use self-consistency comparison but are affected by confirmation biases. To alleviate these, we propose Decompose and Compare Consistency (DeCC) for reliability measurement. By comparing the consistency between the direct answer generated using the VLM's internal reasoning process, and the indirect answers obtained by decomposing the question into sub-questions and reasoning over the sub-answers produced by the VLM, DeCC measures the reliability of VLM's direct answer. Experiments across six vision-language tasks with three VLMs show DeCC's reliability estimation achieves better correlation with task accuracy compared to the existing methods.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 33 references that are in the map. First posted on arXiv 10 Jul 2024.

Builds on

25 of 33 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoning23 Feb
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct

Other references · 23

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  6. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  7. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  8. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  9. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  10. 2017Selective Classification for Deep Neural Networkscited by 799
  11. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  12. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
11 more
  1. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  2. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  3. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  4. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
  5. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  6. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  7. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 209
  8. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 176
  9. 2022Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectlycited by 86
  10. 2023Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMscited by 36
  11. 2024Few-Shot Recalibration of Language Modelscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 3.

7 papers.

  1. 2025Lost in OCR Translation? Vision-Based Approaches to Robust Document Retrievalcited by 10
  2. 2025LogicQA: Logical Anomaly Detection with Vision Language Model Generated Questionscited by 8
  3. 2026DDA-Thinker: Decoupled Dual-Atomic Reinforcement Learning for Reasoning-Driven Image Editingcited by 2
  4. 2024CAST: Cross-modal Alignment Similarity Test for Vision Language Modelscited by 1
  5. 2024Beyond Visual Understanding: Introducing PARROT-360V for Vision Language Model Benchmarkingcited by 0
  6. 2026When Does Consensus Mean Correctness? Measuring the Agreement-Accuracy Coupling with Semantics-Preserving Re-Renderingcited by 0
  7. 2026Consistency Has a Computable Blind Spot: A Commutation Theory of Label-Free Reliability for Vision-Language Figure Readingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Improving VQA Reliability: A Dual-Assessment Approach with Self-Reflection and Cross-Model Verification
  2. 2024Uncertainty-Aware Evaluation for Vision-Language Models
  3. 2024MM-R3: On (In-)Consistency of Vision-Language Models (VLMs)
  4. 2026The Cost of Reasoning: Chain-of-Thought Induces Overconfidence in Vision-Language Models
  5. 2026Towards Annotation-Free Validation of MLLMs: A Vision-Language Logical Consistency Metric
  6. 2026When Vision-Language Models Judge Without Seeing: Exposing Informativeness Bias
  7. 2025Evaluating Variance in Visual Question Answering Benchmarks
  8. 2025Towards Rationale-Answer Alignment of LVLMs via Self-Rationale Calibration