mlsanityData to 13 Oct 2026

Improve Vision Language Model Chain-of-thought Reasoning

arXiv 2410.16198 ↗First posted 21 Oct 2024cs.AI cs.CVcited by 136

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Chain-of-thought (CoT) reasoning in vision language models (VLMs) is crucial for improving interpretability and trustworthiness. However, current training recipes lack robust CoT reasoning data, relying on datasets dominated by short annotations with minimal rationales. In this work, we show that training VLM on short answers does not generalize well to reasoning tasks that require more detailed responses. To address this, we propose a two-fold approach. First, we distill rationales from GPT-4o model to enrich the training data and fine-tune VLMs, boosting their CoT performance. Second, we apply reinforcement learning to further calibrate reasoning quality. Specifically, we construct positive (correct) and negative (incorrect) pairs of model-generated reasoning chains, by comparing their predictions with annotated short answers. Using this pairwise data, we apply the Direct Preference Optimization algorithm to refine the model's reasoning abilities. Our experiments demonstrate significant improvements in CoT reasoning on benchmark datasets and better generalization to direct answer prediction as well. This work emphasizes the importance of incorporating detailed rationales in training and leveraging reinforcement learning to strengthen the reasoning capabilities of VLMs.

Key facts. Cited by 136 papers in the map (data through 13 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 36 of its 47 references that are in the map. First posted on arXiv 21 Oct 2024.

Builds on

36 of 47 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understanding15 Apr
  2. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset22 Feb
  3. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  4. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep
  5. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning19 Mar

Other references · 31

  1. 2024The Llama 3 Herd of Modelscited by 14,486
  2. 2023Visual Instruction Tuningcited by 7,827
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,389
  4. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,692
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,546
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,057
  7. 2019Towards VQA Models That Can Readcited by 1,705
  8. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,295
  9. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,070
  10. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 899
  11. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 827
  12. 2016A Diagram Is Worth A Dozen Imagescited by 796
19 more
  1. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 708
  2. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 706
  3. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 621
  4. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 446
  5. 2021InfographicVQAcited by 431
  6. 2023Detecting and Preventing Hallucinations in Large Vision Language Modelscited by 254
  7. 2024Iterative Reasoning Preference Optimizationcited by 246
  8. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  9. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 239
  10. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 216
  11. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  12. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 146
  13. 2023Silkie: Preference Distillation for Large Visual Language Modelscited by 128
  14. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  15. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 95
  16. 2024Enhancing Large Vision Language Models with Self-Training on Image Comprehensioncited by 70
  17. 2024CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMscited by 49
  18. 2024Step-Controlled DPO: Leveraging Stepwise Error for Enhanced Mathematical Reasoningcited by 35
  19. 2024Notes on Applicability of GPT-4 to Document Understandingcited by 7

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 5; 2025: 87; 2026: 44.

136 papers.

Sort:

By area:

  1. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 659
  2. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 449
  3. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 299
  4. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 178
  5. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 173
  6. 2025LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMscited by 145
  7. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 145
  8. 2025Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Stepcited by 130
  9. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 126
  10. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Modelscited by 114
  11. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 94
  12. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
38 more
  1. 2025More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Modelscited by 85
  2. 2025Unlocking Multimodal Mathematical Reasoning via Process Reward Modelcited by 64
  3. 2025R1-ShareVL: Incentivizing Reasoning Capability of Multimodal Large Language Models via Share-GRPOcited by 52
  4. 2025Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learningcited by 52
  5. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 44
  6. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 43
  7. 2025VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learningcited by 42
  8. 2025Advancing Multimodal Reasoning via Reinforcement Learning with Cold Startcited by 41
  9. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  10. 2025Bring Reason to Vision: Understanding Perception and Reasoning through Model Mergingcited by 40
  11. 2025DriveLMM-o1: A Step-by-Step Reasoning Dataset and Large Multimodal Model for Driving Scenario Understandingcited by 35
  12. 2025Boosting MLLM Reasoning with Text-Debiased Hint-GRPOcited by 32
  13. 2025SafeMLRM: Demystifying Safety in Multi-modal Large Reasoning Modelscited by 29
  14. 2025Seeing the Arrow of Time in Large Multimodal Modelscited by 29
  15. 2025VideoCap-R1: Enhancing MLLMs for Video Captioning via Structured Thinkingcited by 27
  16. 2025Argus: Vision-Centric Reasoning with Grounded Chain-of-Thoughtcited by 26
  17. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
  18. 2025ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMscited by 25
  19. 2025Why Vision Language Models Struggle with Visual Arithmetic? Towards Enhanced Chart and Geometry Understandingcited by 24
  20. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 24
  21. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  22. 2025Mind with Eyes: from Language Reasoning to Multimodal Reasoningcited by 22
  23. 2025A Survey on Agentic Multimodal Large Language Modelscited by 22
  24. 2025Valley2: Exploring Multimodal Models with Scalable Vision-Language Designcited by 21
  25. 2025Tactile-VLA: Unlocking Vision-Language-Action Model's Physical Knowledge for Tactile Generalizationcited by 19
  26. 2025Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Taskscited by 18
  27. 2024AtomThink: Multimodal Slow Thinking with Atomic Step Reasoningcited by 16
  28. 2025Forgotten Polygons: Multimodal Large Language Models are Shape-Blindcited by 16
  29. 2025SiLVR: A Simple Language-based Video Reasoning Frameworkcited by 14
  30. 2025GThinker: Towards General Multimodal Reasoning via Cue-Guided Rethinkingcited by 14
  31. 2025MMReason: An Open-Ended Multi-Modal Multi-Step Reasoning Benchmark for MLLMs Toward AGIcited by 14
  32. 2026When and How Much to Imagine: Adaptive Test-Time Scaling with World Models for Visual Spatial Reasoningcited by 13
  33. 2025Sherlock: Self-Correcting Reasoning in Vision-Language Modelscited by 12
  34. 2025The Synergy Dilemma of Long-CoT SFT and RL: Investigating Post-Training Techniques for Reasoning VLMscited by 12
  35. 2025Corvid: Improving Multimodal Large Language Models Towards Chain-of-Thought Reasoningcited by 11
  36. 2026SceneAlign: Aligning Multimodal Reasoning to Scene Graphs in Complex Visual Scenescited by 11
  37. 2025Can Atomic Step Decomposition Enhance the Self-structured Reasoning of Multimodal Large Models?cited by 10
  38. 2025Infi-MMR: Curriculum-based Unlocking Multimodal Reasoning via Phased Reinforcement Learning in Multimodal Small Language Modelscited by 10

50 most cited of 136 All 136 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning
  2. 2025Latent Chain-of-Thought for Visual Reasoning
  3. 2025Perception Before Reasoning: Two-Stage Reinforcement Learning for Visual Reasoning in Vision-Language Models
  4. 2023Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models
  5. 2025Rethinking RL Scaling for Vision Language Models: A Transparent, From-Scratch Framework and Comprehensive Evaluation Scheme
  6. 2024Vision-Language Models Can Self-Improve Reasoning via Reflection
  7. 2025DeepThinkVLA: Enhancing Reasoning Capability of Vision-Language-Action Models
  8. 2025Cache-of-Thought: Master-Apprentice Framework for Cost-Effective Vision Language Model Reasoning