mlsanityData to 14 Oct 2026

Thinking Before Looking: Improving Multimodal LLM Reasoning via Mitigating Visual Hallucination

arXiv 2411.12591 ↗First posted 15 Nov 2024cs.CV cs.AIcited by 29

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Multimodal large language models (MLLMs) have advanced the integration of visual and linguistic modalities, establishing themselves as the dominant paradigm for visual-language tasks. Current approaches like chain of thought (CoT) reasoning have augmented the cognitive capabilities of large language models (LLMs), yet their adaptation to MLLMs is hindered by heightened risks of hallucination in cross-modality comprehension. In this paper, we find that the thinking while looking paradigm in current multimodal CoT approaches--where reasoning chains are generated alongside visual input--fails to mitigate hallucinations caused by misleading images. To address these limitations, we propose the Visual Inference Chain (VIC) framework, a novel approach that constructs reasoning chains using textual context alone before introducing visual input, effectively reducing cross-modal biases and enhancing multimodal reasoning accuracy. Comprehensive evaluations demonstrate that VIC significantly improves zero-shot performance across various vision-related tasks, mitigating hallucinations while refining the reasoning capabilities of MLLMs. Our code repository can be found at https://github.com/Terry-Xu-666/visual_inference_chain.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 21 of its 48 references that are in the map. First posted on arXiv 15 Nov 2024.

Builds on

21 of 48 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Models22 May
  2. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning25 Mar
  3. 2024CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputs5 Jan
  4. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  5. 2023Multimodal Chain-of-Thought Reasoning in Language Models2 Feb

Other references · 16

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  5. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  6. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  7. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  8. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
  9. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  10. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  11. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 445
  12. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 388
4 more
  1. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 363
  2. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 233
  3. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 226
  4. 2023FaithScore: Fine-grained Evaluations of Hallucinations in Large Vision-Language Modelscited by 46

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 21; 2026: 8.

29 papers.

Sort:

By area:

  1. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 182
  2. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  3. 2025Unlocking Multimodal Mathematical Reasoning via Process Reward Modelcited by 65
  4. 2025First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Trainingcited by 34
  5. 2025SafeMLRM: Demystifying Safety in Multi-modal Large Reasoning Modelscited by 29
  6. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 25
  7. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  8. 2025MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLMcited by 17
  9. 2025Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMscited by 14
  10. 2025OWL: Geometry-Aware Spatial Reasoning for Audio Large Language Modelscited by 10
  11. 2025Aurelia: Test-time Reasoning Distillation in Audio-Visual LLMscited by 9
  12. 2025Mobile-Agent-RAG: Driving Smart Multi-Agent Coordination with Contextual Knowledge Empowerment for Long-Horizon Mobile Automationcited by 5
17 more
  1. 2025WISE: Weak-Supervision-Guided Step-by-Step Explanations for Multimodal LLMs in Image Classificationcited by 4
  2. 2025MEENA (PersianMMMU): Multimodal-Multilingual Educational Exams for N-level Assessmentcited by 3
  3. 2025When to Think and When to Look: Uncertainty-Guided Lookbackcited by 3
  4. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  5. 2025DreamPRM-1.5: Unlocking the Potential of Each Instance for Multimodal Process Reward Model Trainingcited by 1
  6. 2025From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Modelscited by 1
  7. 2025Think First, Assign Next (ThiFAN-VQA): A Two-stage Chain-of-Thought Framework for Post-Disaster Damage Assessmentcited by 1
  8. 2025MMRPT: MultiModal Reinforcement Pre-Training via Masked Vision-Dependent Reasoningcited by 1
  9. 2026Enhancing Video Representations with Spatiotemporal-Semantic Residual to Mitigate Hallucinations in Video Large Multimodal Modelscited by 1
  10. 2026MM-THEBench: Do Reasoning MLLMs Think Reasonably?cited by 1
  11. 2026Decoding Scientific Experimental Images: The SPUR Benchmark for Perception, Understanding, and Reasoningcited by 1
  12. 2026Learn to Think: Improving Multimodal Reasoning through Vision-Aware Self-Improvement Trainingcited by 1
  13. 2026CL-Anomaly: Layer-Adaptive Mixture-of-Experts with Multimodal Large Language Model for Continual Learning in Anomaly Detectioncited by 1
  14. 2025Towards Film-Making Production Dialogue, Narration, Monologue Adaptive Moving Dubbing Benchmarkscited by 0
  15. 2026Dual Tuning for Reasoning Efficacy-Driven Data Curation in Multimodal LLM Trainingcited by 0
  16. 2026A Survey of Multimodal Mathematical Reasoning: From Perception, Alignment to Reasoningcited by 0
  17. 2026Instruct-ICL: Instruction-Guided In-Context Learning for Post-Disaster Damage Assessmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Combating Multimodal LLM Hallucination via Bottom-Up Holistic Reasoning
  2. 2025More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Models
  3. 2026Seeing Through the Chain: Mitigate Hallucination in Multimodal Reasoning Models via CoT Compression and Contrastive Preference Optimization
  4. 2026Understanding and Mitigating Hallucinations in Multimodal Chain-of-Thought Models
  5. 2026HIVE: Understanding Post-Hallucination Reasoning in Vision Language Models
  6. 2026See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs
  7. 2024HaloQuest: A Visual Hallucination Dataset for Advancing Multimodal Reasoning
  8. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Models