mlsanityData to 7 Oct 2026

Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning

arXiv 2403.16999 ↗First posted 25 Mar 2024cs.CVcited by 267

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Multi-Modal Large Language Models (MLLMs) have demonstrated impressive performance in various VQA tasks. However, they often lack interpretability and struggle with complex visual inputs, especially when the resolution of the input image is high or when the interested region that could provide key information for answering the question is small. To address these challenges, we collect and introduce the large-scale Visual CoT dataset comprising 438k question-answer pairs, annotated with intermediate bounding boxes highlighting key regions essential for answering the questions. Additionally, about 98k pairs of them are annotated with detailed reasoning steps. Importantly, we propose a multi-turn processing pipeline that dynamically focuses on visual inputs and provides interpretable thoughts. We also introduce the related benchmark to evaluate the MLLMs in scenarios requiring specific local region identification. Extensive experiments demonstrate the effectiveness of our framework and shed light on better inference strategies. The Visual CoT dataset, benchmark, and pre-trained models are available on https://hao-shao.com/projects/viscot.html to support further research in this area.

Key facts. Cited by 267 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 72 of its 100 references that are in the map. First posted on arXiv 25 Mar 2024.

Builds on

72 of 100 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024CogCoM: A Visual Language Model with Chain-of-Manipulations Reasoning6 Feb
  2. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Models27 Nov
  3. 2023DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models25 Oct
  4. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  6. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action20 Mar
  7. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models8 Mar
  8. 2023Multimodal Chain-of-Thought Reasoning in Language Models2 Feb
  9. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep

Other references · 63

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  12. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
38 more
  1. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  2. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  4. 2019Towards VQA Models That Can Readcited by 1,696
  5. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  6. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  7. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  9. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  10. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  11. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  12. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  13. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  14. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  15. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  16. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  17. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  18. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  19. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  20. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  21. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  22. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  23. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  24. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  25. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  26. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  27. 2021InfographicVQAcited by 430
  28. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  29. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  30. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
  31. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  32. 2022Visual Spatial Reasoningcited by 288
  33. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  34. 2023Generating Images with Multimodal Language Modelscited by 259
  35. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 254
  36. 2023Detecting and Preventing Hallucinations in Large Vision Language Modelscited by 254
  37. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  38. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 24; 2025: 137; 2026: 106.

267 papers.

Sort:

By area:

  1. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 446
  2. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 398
  3. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  4. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
  5. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 171
  6. 2025Thyme: Think Beyond Imagescited by 160
  7. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 144
  8. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 132
  9. 2025Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawingcited by 130
  10. 2025GRIT: Teaching MLLMs to Think with Imagescited by 121
  11. 2025Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokenscited by 120
  12. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
38 more
  1. 2025VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuningcited by 86
  2. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  3. 2025Zebra-CoT: A Dataset for Interleaved Vision Language Reasoningcited by 78
  4. 2025VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learningcited by 68
  5. 2025Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodologycited by 67
  6. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  7. 2025VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perceptioncited by 60
  8. 2025Grounded Chain-of-Thought for Multimodal Large Language Modelscited by 57
  9. 2025ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoningcited by 55
  10. 2024Visual Agents as Fast and Slow Thinkerscited by 53
  11. 2025Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMscited by 48
  12. 2025Ground-R1: Incentivizing Grounded Visual Reasoning via Reinforcement Learningcited by 46
  13. 2025SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoningcited by 45
  14. 2025Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidencecited by 43
  15. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  16. 2024Enhancing Video-LLM Reasoning via Agent-of-Thoughts Distillationcited by 36
  17. 2025When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruningcited by 35
  18. 2025The Homogenizing Effect of Large Language Models on Human Expression and Thoughtcited by 35
  19. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulationcited by 34
  20. 2024F-LMM: Grounding Frozen Large Multimodal Modelscited by 33
  21. 2024Probing the limitations of multimodal language models for chemistry and materials researchcited by 31
  22. 2025VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Modelscited by 31
  23. 2024SketchAgent: Language-Driven Sequential Sketch Generationcited by 30
  24. 2024Thinking Before Looking: Improving Multimodal LLM Reasoning via Mitigating Visual Hallucinationcited by 29
  25. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  26. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  27. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  28. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 26
  29. 2024EasyRef: Omni-Generalized Group Image Reference for Diffusion Models via Multimodal LLMcited by 25
  30. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  31. 2025Mind with Eyes: from Language Reasoning to Multimodal Reasoningcited by 22
  32. 2025ReasonPlan: Unified Scene Prediction and Decision Reasoning for Closed-loop Autonomous Drivingcited by 22
  33. 2025CodePlot-CoT: Mathematical Visual Reasoning by Thinking with Code-Driven Imagescited by 22
  34. 2026Think3D: Thinking with Space for Spatial Reasoningcited by 22
  35. 2025Being-0: A Humanoid Robotic Agent with Vision-Language Models and Modular Skillscited by 21
  36. 2025Toward Generalizable Forgery Detection and Reasoningcited by 20
  37. 2026DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Drivingcited by 20
  38. 2025Multi-Step Visual Reasoning with Visual Tokens Scaling and Verificationcited by 19

50 most cited of 267 All 267 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Visual CoT Makes VLMs Smarter but More Fragile
  2. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
  3. 2026Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoning
  4. 2025Unsupervised Visual Chain-of-Thought Reasoning via Preference Optimization
  5. 2023Towards a Unified Multimodal Reasoning Framework
  6. 2024Advancing Large Multi-modal Models with Explicit Chain-of-Reasoning and Visual Question Generation
  7. 2024M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought
  8. 2024KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoning