mlsanityData to 7 Oct 2026

The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)

arXiv 2309.17421 ↗First posted 29 Sep 2023cs.CV cs.CLcited by 677

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large multimodal models (LMMs) extend large language models (LLMs) with multi-sensory skills, such as visual understanding, to achieve stronger generic intelligence. In this paper, we analyze the latest model, GPT-4V(ision), to deepen the understanding of LMMs. The analysis focuses on the intriguing tasks that GPT-4V can perform, containing test samples to probe the quality and genericity of GPT-4V's capabilities, its supported inputs and working modes, and the effective ways to prompt the model. In our approach to exploring GPT-4V, we curate and organize a collection of carefully designed qualitative samples spanning a variety of domains and tasks. Observations from these samples demonstrate that GPT-4V's unprecedented ability in processing arbitrarily interleaved multimodal inputs and the genericity of its capabilities together make GPT-4V a powerful multimodal generalist system. Furthermore, GPT-4V's unique capability of understanding visual markers drawn on input images can give rise to new human-computer interaction methods such as visual referring prompting. We conclude the report with in-depth discussions on the emerging application scenarios and the future research directions for GPT-4V-based systems. We hope that this preliminary exploration will inspire future research on the next-generation multimodal task formulation, new ways to exploit and enhance LMMs to solve real-world problems, and gaining better understanding of multimodal foundation models. Finally, we acknowledge that the model under our study is solely the product of OpenAI's innovative work, and they should be fully credited for its development. Please see the GPT-4V contributions paper for the authorship and credit attribution: https://cdn.openai.com/contributions/gpt-4v.pdf

Key facts. Cited by 677 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 153 references that are in the map. First posted on arXiv 29 Sep 2023.

Builds on

47 of 153 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 46

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  7. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  9. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  10. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
34 more
  1. 2022Emergent Abilities of Large Language Modelscited by 2,483
  2. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  3. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  4. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  5. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  6. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  7. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  8. 2023PaLM 2 Technical Reportcited by 1,154
  9. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  10. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  11. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  12. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  13. 2022A Survey on In-context Learningcited by 717
  14. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  15. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  16. 2023Segment Everything Everywhere All at Oncecited by 546
  17. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  18. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  19. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  20. 2023Augmented Language Models: a Surveycited by 398
  21. 2023Language Models can Solve Computer Taskscited by 371
  22. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  23. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  24. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  25. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  26. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  27. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  28. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  29. 2023ART: Automatic multi-step reasoning and tool-use for large language modelscited by 193
  30. 2023TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIscited by 184
  31. 2022GRiT: A Generative Region-to-text Transformer for Object Understandingcited by 119
  32. 2021Broaden the Vision: Geo-Diverse Visual Commonsense Reasoningcited by 59
  33. 2020Point and Ask: Incorporating Pointing into Visual Question Answeringcited by 39
  34. 2023Bot or Human? Detecting ChatGPT Imposters with A Single Questioncited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 94; 2024: 326; 2025: 188; 2026: 69.

677 papers.

Sort:

By area:

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  5. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  6. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  7. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  8. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  9. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  10. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  11. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  12. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
38 more
  1. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  2. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  3. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 446
  4. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 424
  5. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  6. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  7. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 398
  8. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  9. 2024TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policiescited by 298
  10. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 281
  11. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  12. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  13. 2023G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Modelcited by 208
  14. 2023VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluationcited by 196
  15. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 193
  16. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 192
  17. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  18. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  19. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 167
  20. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  21. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 160
  22. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generationcited by 158
  23. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 156
  24. 2024GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generationcited by 151
  25. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuningcited by 149
  26. 2024Tree Search for Language Model Agentscited by 148
  27. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 147
  28. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 146
  29. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 144
  30. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 144
  31. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Modelscited by 138
  32. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  33. 2023OpenAgents: An Open Platform for Language Agents in the Wildcited by 136
  34. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  35. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 135
  36. 2024StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understandingcited by 135
  37. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  38. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 130

50 most cited of 677 All 677 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Lost in Translation: When GPT-4V(ision) Can't See Eye to Eye with Text. A Vision-Language-Consistency Analysis of VLLMs and Beyond
  2. 2024Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiency
  3. 2023NExT-GPT: Any-to-Any Multimodal LLM
  4. 2023Large Multimodal Models: Notes on CVPR 2023 Tutorial
  5. 2023Multimodal Analysis Of Google Bard And GPT-Vision: Experiments In Visual Reasoning
  6. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
  7. 2023Assessing GPT4-V on Structured Reasoning Tasks
  8. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction