mlsanityData to 7 Oct 2026

Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning

arXiv 2401.06805 ↗First posted 10 Jan 2024cs.CL cs.AIcited by 136

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Strong Artificial Intelligence (Strong AI) or Artificial General Intelligence (AGI) with abstract reasoning ability is the goal of next-generation AI. Recent advancements in Large Language Models (LLMs), along with the emerging field of Multimodal Large Language Models (MLLMs), have demonstrated impressive capabilities across a wide range of multimodal tasks and applications. Particularly, various MLLMs, each with distinct model architectures, training data, and training stages, have been evaluated across a broad range of MLLM benchmarks. These studies have, to varying degrees, revealed different aspects of the current capabilities of MLLMs. However, the reasoning abilities of MLLMs have not been systematically investigated. In this survey, we comprehensively review the existing evaluation protocols of multimodal reasoning, categorize and illustrate the frontiers of MLLMs, introduce recent trends in applications of MLLMs on reasoning-intensive tasks, and finally discuss current practices and future directions. We believe our survey establishes a solid base and sheds light on this important topic, multimodal reasoning.

Key facts. Cited by 136 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 131 of its 179 references that are in the map. First posted on arXiv 10 Jan 2024.

Builds on

131 of 179 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  2. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action20 Mar
  3. 2022Visual Programming: Compositional visual reasoning without training18 Nov
  4. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep
  5. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language1 Apr
  6. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning19 Mar

Other references · 125

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2017Proximal Policy Optimization Algorithmscited by 14,441
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  9. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  10. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
38 more
  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  12. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  13. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  14. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  15. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  16. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  17. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  18. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  19. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  20. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  21. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  22. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  23. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  24. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  25. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  26. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  27. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  28. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  29. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  30. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  31. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  32. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  33. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  34. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  35. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  36. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  37. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  38. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143

50 most cited of 125 All 125 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 36; 2025: 80; 2026: 20.

136 papers.

Sort:

By area:

  1. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 244
  2. 2024The BrowserGym Ecosystem for Web Agent Researchcited by 152
  3. 2025Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters Morecited by 149
  4. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  5. 2024Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoningcited by 83
  6. 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
  7. 2024MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Modelscited by 51
  8. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  9. 2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Modelscited by 35
  10. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  11. 2025MedFrameQA: A Multi-Image Medical VQA Benchmark for Clinical Reasoningcited by 33
  12. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 31
38 more
  1. 2025MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Researchcited by 30
  2. 2024ALCM: Autonomous LLM-Augmented Causal Discovery Frameworkcited by 29
  3. 2025100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Modelscited by 29
  4. 2025Shifting AI Efficiency From Model-Centric to Data-Centric Compressioncited by 29
  5. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  6. 2024Facial Affective Behavior Analysis with Instruction Tuningcited by 26
  7. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 26
  8. 2025AStar: Boosting Multimodal Reasoning with Automated Structured Thinkingcited by 25
  9. 2024Remote Sensing SpatioTemporal Vision-Language Models: A Comprehensive Surveycited by 22
  10. 2025Are Multimodal Large Language Models Ready for Omnidirectional Spatial Reasoning?cited by 22
  11. 2025Efficient Multi-modal Large Language Models via Progressive Consistency Distillationcited by 22
  12. 2024See it, Think it, Sorted: Large Multimodal Models are Few-shot Time Series Anomaly Analyzerscited by 20
  13. 2025VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoocited by 20
  14. 2024SMoLoRA: Exploring and Defying Dual Catastrophic Forgetting in Continual Visual Instruction Tuningcited by 19
  15. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 19
  16. 2025SurveillanceVQA-589K: A Benchmark for Comprehensive Surveillance Video-Language Understanding with Large Modelscited by 18
  17. 2025THINK-Bench: Evaluating Thinking Efficiency and Chain-of-Thought Quality of Large Reasoning Modelscited by 18
  18. 2024From the Least to the Most: Building a Plug-and-Play Visual Reasoner via Data Synthesiscited by 17
  19. 2024EAGLE: Elevating Geometric Reasoning through LLM-empowered Visual Instruction Tuningcited by 16
  20. 2024Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?cited by 16
  21. 2024ExpLLM: Towards Chain of Thought for Facial Expression Recognitioncited by 15
  22. 2025PuzzleBench: A Fully Dynamic Evaluation Framework for Large Multimodal Models on Puzzle Solvingcited by 15
  23. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  24. 2024MobA: Multifaceted Memory-Enhanced Adaptive Planning for Efficient Mobile Task Automationcited by 14
  25. 2025MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMscited by 14
  26. 2025LoRA in LoRA: Towards Parameter-Efficient Architecture Expansion for Continual Visual Instruction Tuningcited by 13
  27. 2025Reinforced Reasoning for Embodied Planningcited by 12
  28. 2025Learning from Diverse Reasoning Paths with Routing and Collaborationcited by 12
  29. 2024InfiMM-HD: A Leap Forward in High-Resolution Multimodal Understandingcited by 11
  30. 2025Does Chain-of-Thought Reasoning Help Mobile GUI Agent? An Empirical Studycited by 11
  31. 2025Weaving Context Across Images: Improving Vision-Language Models through Focus-Centric Visual Chainscited by 11
  32. 2026MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chainscited by 11
  33. 2024Data-Juicer Sandbox: A Feedback-Driven Suite for Multimodal Data-Model Co-developmentcited by 10
  34. 2025MAPS: Advancing Multi-Modal Reasoning in Expert-Level Physical Sciencecited by 10
  35. 2025MMSciBench: Benchmarking Language Models on Chinese Multimodal Scientific Problemscited by 10
  36. 2025MathFlow: Enhancing the Perceptual Flow of MLLMs for Visual Mathematical Problemscited by 10
  37. 2025Benchmarking Multimodal Mathematical Reasoning with Explicit Visual Dependencycited by 10
  38. 2025Why 1 + 1 < 1 in Visual Token Pruning: Beyond Naive Integration via Multi-Objective Balanced Coveringcited by 10

50 most cited of 136 All 136 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Models
  2. 2025MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models
  3. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs
  4. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark
  5. 2023A Survey on Multimodal Large Language Models
  6. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
  7. 2023MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria
  8. 2024Surveying the MLLM Landscape: A Meta-Review of Current Surveys