mlsanityData to 7 Oct 2026

LLaVA-CoT: Let Vision Language Models Reason Step-by-Step

arXiv 2411.10440 ↗First posted 15 Nov 2024cs.CVcited by 446

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Large language models have demonstrated substantial advancements in reasoning capabilities. However, current Vision-Language Models (VLMs) often struggle to perform systematic and structured reasoning, especially when handling complex visual question-answering tasks. In this work, we introduce LLaVA-CoT, a large VLM designed to conduct autonomous multistage reasoning. Unlike chain-of-thought prompting, LLaVA-CoT independently engages in sequential stages of summarization, visual interpretation, logical reasoning, and conclusion generation. This structured approach enables LLaVA-CoT to achieve marked improvements on reasoning-intensive tasks. To accomplish this, we construct the LLaVA-CoT-100k dataset, integrating samples from various visual question answering sources and providing structured reasoning annotations. Besides, we propose a test-time stage-wise retracing search method (SWIRES), which enables effective and efficient test-time scaling. Remarkably, with only 100k training samples and test-time scaling, LLaVA-CoT not only outperforms its base model by 9.4% on a wide range of multimodal reasoning benchmarks, but also surpasses the performance of larger and even closed-source models, such as Gemini-1.5-pro, GPT-4o-mini, and Llama-3.2-90B-Vision-Instruct. The code, dataset, and pre-trained weights are publicly available at https://github.com/PKU-YuanGroup/LLaVA-CoT.

Key facts. Cited by 446 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 59 of its 76 references that are in the map. First posted on arXiv 15 Nov 2024.

Builds on

59 of 76 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Improve Vision Language Model Chain-of-thought Reasoning21 Oct
  2. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning25 Mar
  3. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  5. 2022Visual Programming: Compositional visual reasoning without training18 Nov
  6. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep
  7. 2022CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoning10 Aug
  8. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning19 Mar
  9. 2022A Review of Emerging Research Directions in Abstract Visual Reasoning21 Feb

Other references · 50

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2024GPT-4o System Cardcited by 4,467
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  9. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  10. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  11. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  12. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
38 more
  1. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  2. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  3. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  4. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  5. 2016A Diagram Is Worth A Dozen Imagescited by 793
  6. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  7. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  8. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  9. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  10. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  11. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  12. 2022Large Language Models Can Self-Improvecited by 656
  13. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  14. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  15. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  16. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  17. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 304
  18. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  19. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 170
  20. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 167
  21. 2024O1 Replication Journey: A Strategic Progress Report -- Part 1cited by 140
  22. 2024Evaluation of OpenAI o1: Opportunities and Challenges of AGIcited by 127
  23. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluationcited by 123
  24. 2023Deductive Verification of Chain-of-Thought Reasoningcited by 122
  25. 2022CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailmentcited by 113
  26. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  27. 2024A Comparative Study on Reasoning Patterns of OpenAI's o1 Modelcited by 62
  28. 2024EvaGaussians: Event Stream Assisted Gaussian Splatting from Blurry Imagescited by 56
  29. 2024Variational Best-of-N Alignmentcited by 49
  30. 2024Rethinking Data Selection for Supervised Fine-Tuningcited by 35
  31. 2024Prism: A Framework for Decoupling and Assessing the Capabilities of VLMscited by 32
  32. 2022Toward 3D Spatial Reasoning for Human-like Text-based Visual Question Answeringcited by 24
  33. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunitiescited by 24
  34. 2023TEILP: Time Prediction over Knowledge Graphs via Logical Reasoningcited by 23
  35. 2024Fine-tuning Large Language Models with Sequential Instructionscited by 20
  36. 2024Towards Neuro-Symbolic Video Understandingcited by 16
  37. 2022Joint learning of object graph and relation graph for visual question answeringcited by 13
  38. 2024Multimodal Structured Generation: CVPR's 2nd MMFM Challenge Technical Reportcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 294; 2026: 140.

446 papers.

Sort:

By area:

  1. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 653
  2. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  3. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalizationcited by 364
  4. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  5. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 296
  6. 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learningcited by 267
  7. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 257
  8. 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Modelscited by 209
  9. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  10. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  11. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 171
  12. 2025R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Modelcited by 163
38 more
  1. 2025LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMscited by 145
  2. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 144
  3. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  4. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 125
  5. 2025OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cyclescited by 122
  6. 2025SoftCoT: Soft Chain-of-Thought for Efficient Reasoning with LLMscited by 110
  7. 2025Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?cited by 107
  8. 2024ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Explorationcited by 99
  9. 2025Latent Visual Reasoningcited by 99
  10. 2025Self-Rewarding Vision-Language Model via Reasoning Decompositioncited by 95
  11. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 94
  12. 2025Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Modelscited by 92
  13. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  14. 2025VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuningcited by 86
  15. 2025Visionary-R1: Mitigating Shortcuts in Visual Reasoning with Reinforcement Learningcited by 86
  16. 2025VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Usecited by 84
  17. 2025More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Modelscited by 84
  18. 2025UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learningcited by 83
  19. 2025Zebra-CoT: A Dataset for Interleaved Vision Language Reasoningcited by 78
  20. 2025Efficient Reasoning with Hidden Thinkingcited by 72
  21. 2025VGR: Visual Grounded Reasoningcited by 70
  22. 2025Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoningcited by 67
  23. 2025Reinforcement Learning Outperforms Supervised Fine-Tuning: A Case Study on Audio Question Answeringcited by 64
  24. 2025VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perceptioncited by 60
  25. 2025LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Callingcited by 58
  26. 2025Grounded Chain-of-Thought for Multimodal Large Language Modelscited by 57
  27. 2025Look-Back: Implicit Visual Re-focusing in MLLM Reasoningcited by 54
  28. 2025SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learningcited by 53
  29. 2025Uni-cot: Towards Unified Chain-of-Thought Reasoning Across Text and Visioncited by 52
  30. 2025Improved Visual-Spatial Reasoning via R1-Zero-Like Trainingcited by 51
  31. 2025UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobilitycited by 48
  32. 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?cited by 48
  33. 2025Ground-R1: Incentivizing Grounded Visual Reasoning via Reinforcement Learningcited by 46
  34. 2025Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operationscited by 46
  35. 2025SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoningcited by 45
  36. 2025Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Rewardcited by 45
  37. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 43
  38. 2025Implicit Reasoning in Large Language Models: A Comprehensive Surveycited by 42

50 most cited of 446 All 446 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Elevating Visual Question Answering through Implicitly Learned Reasoning Pathways in LVLMs
  2. 2025Cache-of-Thought: Master-Apprentice Framework for Cost-Effective Vision Language Model Reasoning
  3. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models
  4. 2025Visual CoT Makes VLMs Smarter but More Fragile
  5. 2023Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questions
  6. 2023Large Language Models are Visual Reasoning Coordinators
  7. 2025V-REX: Benchmarking Exploratory Visual Reasoning via Chain-of-Questions
  8. 2024Enhancing Advanced Visual Reasoning Ability of Large Language Models