mlsanityData to 7 Oct 2026

Generalist Multimodal AI: A Review of Architectures, Challenges and Opportunities

arXiv 2406.05496 ↗First posted 8 Jun 2024cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Multimodal models are expected to be a critical component to future advances in artificial intelligence. This field is starting to grow rapidly with a surge of new design elements motivated by the success of foundation models in natural language processing (NLP) and vision. It is widely hoped that further extending the foundation models to multiple modalities (e.g., text, image, video, sensor, time series, graph, etc.) will ultimately lead to generalist multimodal models, i.e. one model across different data modalities and tasks. However, there is little research that systematically analyzes recent multimodal models (particularly the ones that work beyond text and vision) with respect to the underling architecture proposed. Therefore, this work provides a fresh perspective on generalist multimodal models (GMMs) via a novel architecture and training configuration specific taxonomy. This includes factors such as Unifiability, Modularity, and Adaptability that are pertinent and essential to the wide adoption and application of GMMs. The review further highlights key challenges and prospects for the field and guide the researchers into the new advancements.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 78 of its 101 references that are in the map. First posted on arXiv 8 Jun 2024.

Builds on

78 of 101 references.

References · 78

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  10. 2020Scaling Laws for Neural Language Modelscited by 6,411
  11. 2017Neural Discrete Representation Learningcited by 4,650
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
38 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  4. 2023Mistral 7Bcited by 3,065
  5. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,141
  6. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  7. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  8. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  9. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  10. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  11. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  13. 2019How Can We Know What Language Models Know?cited by 953
  14. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  15. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  16. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  17. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  18. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  19. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  20. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  21. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  22. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
  23. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  24. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 479
  25. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  26. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  27. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  28. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 364
  29. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  30. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 316
  31. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  32. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  33. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  34. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  35. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  36. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 167
  37. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  38. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Revisit the Imbalance Optimization in Multi-task Learning: An Experimental Analysiscited by 3
  2. 2025Hierarchical Vision-Language Alignment for Text-to-Image Generation via Diffusion Modelscited by 0
  3. 2026MULTISEISMO: A Multimodal Seismic Dataset and Model for Cross-Modal Seismic Understandingcited by 0