mlsanityData to 14 Oct 2026

MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action

arXiv 2303.11381 ↗First posted 20 Mar 2023cs.CV cs.CL cs.LGcited by 517

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

We propose MM-REACT, a system paradigm that integrates ChatGPT with a pool of vision experts to achieve multimodal reasoning and action. In this paper, we define and explore a comprehensive list of advanced vision tasks that are intriguing to solve, but may exceed the capabilities of existing vision and vision-language models. To achieve such advanced visual intelligence, MM-REACT introduces a textual prompt design that can represent text descriptions, textualized spatial coordinates, and aligned file names for dense visual signals such as images and videos. MM-REACT's prompt design allows language models to accept, associate, and process multimodal information, thereby facilitating the synergetic combination of ChatGPT and various vision experts. Zero-shot experiments demonstrate MM-REACT's effectiveness in addressing the specified capabilities of interests and its wide application in different scenarios that require advanced visual understanding. Furthermore, we discuss and compare MM-REACT's system paradigm with an alternative approach that extends language models for multimodal scenarios through joint finetuning. Code, demo, video, and visualization are available at https://multimodal-react.github.io/

Key facts. Cited by 517 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 35 of its 45 references that are in the map. First posted on arXiv 20 Mar 2023.

Builds on

35 of 45 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023ViperGPT: Visual Inference via Python Execution for Reasoning14 Mar
  2. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models8 Mar
  3. 2023Multimodal Chain-of-Thought Reasoning in Language Models2 Feb
  4. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep
  5. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language1 Apr

Other references · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  8. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  9. 2020End-to-End Object Detection with Transformerscited by 7,414
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
18 more
  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  3. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  4. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  5. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  7. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  8. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 923
  9. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  10. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  11. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  12. 2022PAL: Program-aided Language Modelscited by 652
  13. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  14. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  15. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 301
  16. 2022Generalized Decoding for Pixel, Image, and Languagecited by 294
  17. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  18. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 138; 2024: 138; 2025: 128; 2026: 113.

517 papers.

Sort:

By area:

  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  4. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  5. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  7. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  8. 2023A Survey on Multimodal Large Language Modelscited by 909
  9. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 876
  10. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  11. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  12. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
38 more
  1. 2023A Comprehensive Overview of Large Language Modelscited by 648
  2. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 631
  3. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  4. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  5. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 551
  6. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  7. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  8. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  9. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  10. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 363
  11. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
  12. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 347
  13. 2023Large Language Models as Tool Makerscited by 303
  14. 2023Tool Learning with Foundation Modelscited by 302
  15. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  16. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 299
  17. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 297
  18. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 293
  19. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 286
  20. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 284
  21. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 280
  22. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 279
  23. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  24. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 274
  25. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 268
  26. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  27. 2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problemscited by 260
  28. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 248
  29. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 234
  30. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  31. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 229
  32. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 226
  33. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  34. 2024Tool Learning with Large Language Models: A Surveycited by 223
  35. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 218
  36. 2025Imagine while Reasoning in Space: Multimodal Visualization-of-Thoughtcited by 206
  37. 2023Multimodal Large Language Models: A Surveycited by 195
  38. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 194

50 most cited of 517 All 517 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts
  2. 2024VisionGPT: Vision-Language Understanding Agent Using Generalized Multimodal Framework
  3. 2025Mind with Eyes: from Language Reasoning to Multimodal Reasoning
  4. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI
  5. 2025Argus: Vision-Centric Reasoning with Grounded Chain-of-Thought
  6. 2026MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods
  7. 2023Multimodal Analysis Of Google Bard And GPT-Vision: Experiments In Visual Reasoning
  8. 2024POEM: Interactive Prompt Optimization for Enhancing Multimodal Reasoning of Large Language Models