mlsanityData to 14 Oct 2026

Multimodal Chain-of-Thought Reasoning in Language Models

arXiv 2302.00923 ↗First posted 2 Feb 2023cs.CL cs.AI cs.CVcited by 752

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Large language models (LLMs) have shown impressive performance on complex reasoning by leveraging chain-of-thought (CoT) prompting to generate intermediate reasoning chains as the rationale to infer the answer. However, existing CoT studies have primarily focused on the language modality. We propose Multimodal-CoT that incorporates language (text) and vision (images) modalities into a two-stage framework that separates rationale generation and answer inference. In this way, answer inference can leverage better generated rationales that are based on multimodal information. Experimental results on ScienceQA and A-OKVQA benchmark datasets show the effectiveness of our proposed approach. With Multimodal-CoT, our model under 1 billion parameters achieves state-of-the-art performance on the ScienceQA benchmark. Our analysis indicates that Multimodal-CoT offers the advantages of mitigating hallucination and enhancing convergence speed. Code is publicly available at https://github.com/amazon-science/mm-cot.

Key facts. Cited by 752 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 48 of its 66 references that are in the map. First posted on arXiv 2 Feb 2023.

Builds on

48 of 66 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning12 Jan
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep

Other references · 46

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  8. 2020End-to-End Object Detection with Transformerscited by 7,414
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  12. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
34 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  3. 2022Emergent Abilities of Large Language Modelscited by 2,512
  4. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  5. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  6. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  7. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  8. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  9. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  10. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  11. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  12. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  13. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  14. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
  15. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  16. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
  17. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
  18. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
  19. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 388
  20. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 350
  21. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 339
  22. 2022Large Language Models Are Reasoning Teacherscited by 329
  23. 2022Teaching Small Language Models to Reasoncited by 284
  24. 2018Bilinear Attention Networkscited by 280
  25. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 267
  26. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  27. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  28. 2018Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answeringcited by 114
  29. 2022Rationale-Augmented Ensembles in Language Modelscited by 111
  30. 2022Language Models are General-Purpose Interfacescited by 97
  31. 2022Iteratively Prompt Pre-trained Language Models for Chain of Thoughtcited by 75
  32. 2021Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translationcited by 44
  33. 2022On Vision Features in Multimodal Machine Translationcited by 31
  34. 2023Universal Multimodal Representation for Language Understandingcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 114; 2024: 162; 2025: 273; 2026: 203.

752 papers.

Sort:

By area:

  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2023A Survey of Large Language Modelscited by 2,576
  3. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  4. 2023A Survey on Multimodal Large Language Modelscited by 909
  5. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  6. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  7. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  8. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  9. 2023A Comprehensive Overview of Large Language Modelscited by 648
  10. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  11. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  12. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 484
38 more
  1. 2023Augmented Language Models: a Surveycited by 403
  2. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  3. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
  4. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 354
  5. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 347
  6. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  7. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 280
  8. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  9. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 268
  10. 2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problemscited by 260
  11. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 256
  12. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 231
  13. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 224
  14. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  15. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  16. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 199
  17. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 197
  18. 2025MolmoAct: Action Reasoning Models that can Reason in Spacecited by 197
  19. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  20. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 185
  21. 2023Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Modelscited by 184
  22. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 182
  23. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 175
  24. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 172
  25. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  26. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 164
  27. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 159
  28. 2024Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognitioncited by 153
  29. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 145
  30. 2025Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawingcited by 136
  31. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 134
  32. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  33. 2025Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Stepcited by 131
  34. 2023DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Modelscited by 128
  35. 2025GRIT: Teaching MLLMs to Think with Imagescited by 125
  36. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformationcited by 124
  37. 2025OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cyclescited by 124
  38. 2025Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokenscited by 124

50 most cited of 752 All 752 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Cross Domain Evaluation of Multimodal Chain-of-Thought Reasoning of different datasets into the Amazon CoT Framework
  2. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
  3. 2025Chain of Questions: Guiding Multimodal Curiosity in Language Models
  4. 2024Vision-Language Models Can Self-Improve Reasoning via Reflection
  5. 2023Multi-modal Latent Space Learning for Chain-of-Thought Reasoning in Language Models
  6. 2025Grounded Chain-of-Thought for Multimodal Large Language Models
  7. 2024M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought
  8. 2024Fine-Tuning on Diverse Reasoning Chains Drives Within-Inference CoT Refinement in LLMs