mlsanityData to 14 Oct 2026

Multimodal Foundation Models: From Specialists to General-Purpose Assistants

arXiv 2309.10020 ↗First posted 18 Sep 2023cs.CV cs.CLcited by 247

Not on the map. Why?

Cited by 82 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper presents a comprehensive survey of the taxonomy and evolution of multimodal foundation models that demonstrate vision and vision-language capabilities, focusing on the transition from specialist models to general-purpose assistants. The research landscape encompasses five core topics, categorized into two classes. (i) We start with a survey of well-established research areas: multimodal foundation models pre-trained for specific purposes, including two topics -- methods of learning vision backbones for visual understanding and text-to-image generation. (ii) Then, we present recent advances in exploratory, open research areas: multimodal foundation models that aim to play the role of general-purpose assistants, including three topics -- unified vision models inspired by large language models (LLMs), end-to-end training of multimodal LLMs, and chaining multimodal tools with LLMs. The target audiences of the paper are researchers, graduate students, and professionals in computer vision and vision-language multimodal communities who are eager to learn the basics and recent advances in multimodal foundation models.

Key facts. Cited by 247 papers in the map (data through 14 Oct 2026). Builds on 263 of its 568 references that are in the map. First posted on arXiv 18 Sep 2023.

Builds on

263 of 568 references.

References · 263

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2023Segment Anythingcited by 8,234
  9. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  10. 2023Visual Instruction Tuningcited by 7,869
  11. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  12. 2020Scaling Laws for Neural Language Modelscited by 6,586
38 more
  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2017Neural Discrete Representation Learningcited by 4,712
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  11. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  12. 2013Auto-Encoding Variational Bayescited by 4,053
  13. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  14. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  15. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  16. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  17. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  18. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  19. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  20. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  21. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  22. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  23. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  24. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  25. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  26. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,161
  27. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  28. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  29. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  30. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  31. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  32. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  33. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  34. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  35. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  36. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  37. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  38. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,351

50 most cited of 263 All 263 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 38; 2024: 106; 2025: 77; 2026: 26.

247 papers.

Sort:

By area:

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  3. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  4. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  5. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  6. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
  7. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  8. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  9. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  10. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 299
  11. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 250
  12. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 232
38 more
  1. 2023Video Understanding with Large Language Models: A Surveycited by 206
  2. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 193
  3. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 176
  4. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 176
  5. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 163
  6. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 159
  7. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  8. 2023Q-Instruct: Improving Low-level Visual Abilities for Multi-modality Foundation Modelscited by 152
  9. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 145
  10. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 144
  11. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 139
  12. 2024LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Modelcited by 138
  13. 2023OpenAgents: An Open Platform for Language Agents in the Wildcited by 137
  14. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  15. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 134
  16. 2023Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordiacited by 126
  17. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmarkcited by 120
  18. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 113
  19. 2024Why Tabular Foundation Models Should Be a Research Prioritycited by 106
  20. 2025On the Theoretical Limitations of Embedding-Based Retrievalcited by 104
  21. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 102
  22. 2024Efficient Multimodal Large Language Models: A Surveycited by 96
  23. 2025LaViDa: A Large Diffusion Language Model for Multimodal Understandingcited by 84
  24. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 83
  25. 2024Detecting Multimedia Generated by Large AI Models: A Surveycited by 76
  26. 2024SNIFFER: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detectioncited by 74
  27. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 73
  28. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
  29. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 66
  30. 2023Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorldcited by 65
  31. 2023Large Language Models are Visual Reasoning Coordinatorscited by 58
  32. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  33. 2025The Path Not Taken: RLVR Provably Learns Off the Principalscited by 56
  34. 2024AnomalyDINO: Boosting Patch-based Few-shot Anomaly Detection with DINOv2cited by 53
  35. 2023Visual In-Context Promptingcited by 52
  36. 2023Continual Instruction Tuning for Large Multimodal Modelscited by 49
  37. 2023LLaVA-Interactive: An All-in-One Demo for Image Chat, Segmentation, Generation and Editingcited by 47
  38. 2024How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Promptscited by 47

50 most cited of 247 All 247 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Large Multimodal Models: Notes on CVPR 2023 Tutorial
  2. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  3. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Models
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
  5. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  6. 2024VisionGPT: Vision-Language Understanding Agent Using Generalized Multimodal Framework
  7. 2024An Introduction to Vision-Language Modeling
  8. 2023Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistants