mlsanityData to 7 Oct 2026

Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey

arXiv 2302.10035 ↗First posted 20 Feb 2023cs.CV cs.AI cs.MMcited by 153

Not on the map. Why?

Cited by 31 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

With the urgent demand for generalized deep models, many pre-trained big models are proposed, such as BERT, ViT, GPT, etc. Inspired by the success of these models in single domains (like computer vision and natural language processing), the multi-modal pre-trained big models have also drawn more and more attention in recent years. In this work, we give a comprehensive survey of these models and hope this paper could provide new insights and helps fresh researchers to track the most cutting-edge works. Specifically, we firstly introduce the background of multi-modal pre-training by reviewing the conventional deep learning, pre-training works in natural language process, computer vision, and speech. Then, we introduce the task definition, key challenges, and advantages of multi-modal pre-training models (MM-PTMs), and discuss the MM-PTMs with a focus on data, objectives, network architectures, and knowledge enhanced pre-training. After that, we introduce the downstream tasks used for the validation of large-scale MM-PTMs, including generative, classification, and regression tasks. We also give visualization and analysis of the model parameters and results on representative downstream tasks. Finally, we point out possible research directions for this topic that may benefit future works. In addition, we maintain a continuously updated paper list for large-scale pre-trained multi-modal big models: https://github.com/wangxiao5791509/MultiModal_BigModels_Survey. This paper has been published by the journal Machine Intelligence Research (MIR), https://link.springer.com/article/10.1007/s11633-022-1410-8, DOI: 10.1007/s11633-022-1410-8, vol. 20, no. 4, pp. 447-482, 2023.

Key facts. Cited by 153 papers in the map (data through 7 Oct 2026). Builds on 205 of its 292 references that are in the map. First posted on arXiv 20 Feb 2023.

Builds on

205 of 292 references.

References · 205

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  9. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  10. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  11. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  12. 2017Graph Attention Networkscited by 8,854
38 more
  1. 2020End-to-End Object Detection with Transformerscited by 7,384
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  3. 2016Densely Connected Convolutional Networkscited by 6,691
  4. 2017Inductive Representation Learning on Large Graphscited by 6,284
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  13. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  14. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  15. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  16. 2015VQA: Visual Question Answeringcited by 2,489
  17. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  18. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  19. 2018Deep contextualized word representationscited by 2,285
  20. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  21. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  22. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  23. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  24. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  25. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,684
  26. 2022Visual Prompt Tuningcited by 1,674
  27. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  28. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  29. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  30. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
  31. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  32. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  33. 2016Variational Graph Auto-Encoderscited by 1,373
  34. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  35. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  36. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  37. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  38. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224

50 most cited of 205 All 205 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 42; 2024: 61; 2025: 37; 2026: 13.

153 papers.

Sort:

By area:

  1. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  2. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  3. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  4. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  5. 2023Multimodal Large Language Models: A Surveycited by 193
  6. 2023TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Seriescited by 165
  7. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 156
  8. 2023A Survey on Graph Neural Networks for Time Series: Forecasting, Classification, Imputation, and Anomaly Detectioncited by 145
  9. 2023Large Selective Kernel Network for Remote Sensing Object Detectioncited by 98
  10. 2024Segment Anything Model for Medical Image Segmentation: Current Applications and Future Directionscited by 97
  11. 2024Merging Multi-Task Models via Weight-Ensembling Mixture of Expertscited by 85
  12. 2023Review of Large Vision Models and Visual Prompt Engineeringcited by 77
38 more
  1. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  2. 2023GPTEval: A Survey on Assessments of ChatGPT and GPT-4cited by 73
  3. 2023UniPAD: A Universal Pre-training Paradigm for Autonomous Drivingcited by 73
  4. 2024MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistancecited by 72
  5. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 66
  6. 2023A Comprehensive Survey on Segment Anything Model for Vision and Beyondcited by 64
  7. 2023AI-Generated Content (AIGC): A Surveycited by 59
  8. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  9. 2024Toward Visual Grounding: A Surveycited by 52
  10. 2023Improving Zero-Shot Generalization for CLIP with Synthesized Promptscited by 51
  11. 2024Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoningcited by 50
  12. 2023UniDream: Unifying Diffusion Priors for Relightable Text-to-3D Generationcited by 47
  13. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  14. 2023Exploring the Landscape of Machine Unlearning: A Comprehensive Survey and Taxonomycited by 44
  15. 2024LSKNet: A Foundation Lightweight Backbone for Remote Sensingcited by 44
  16. 2023SNIP: Bridging Mathematical Symbolic and Numeric Realms with Unified Pre-trainingcited by 41
  17. 2023LLM4EDA: Emerging Progress in Large Language Models for Electronic Design Automationcited by 40
  18. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  19. 2024Visual Adversarial Attack on Vision-Language Models for Autonomous Drivingcited by 35
  20. 2025Every SAM Drop Counts: Embracing Semantic Priors for Multi-Modality Image Fusion and Beyondcited by 34
  21. 2024CXPMRG-Bench: Pre-training and Benchmarking for X-ray Medical Report Generation on CheXpert Plus Datasetcited by 33
  22. 2025R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuningcited by 33
  23. 2025MV-MATH: Evaluating Multimodal Math Reasoning in Multi-Visual Contextscited by 32
  24. 2023CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservationcited by 29
  25. 2024Continual Forgetting for Pre-trained Vision Modelscited by 29
  26. 2024Heterogeneous Contrastive Learning for Foundation Models and Beyondcited by 27
  27. 2023Pedestrian Attribute Recognition via CLIP based Prompt Vision-Language Fusioncited by 24
  28. 2024Generative Multi-modal Models are Good Class-Incremental Learnerscited by 22
  29. 2024Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Mergingcited by 22
  30. 2023Towards Segment Anything Model (SAM) for Medical Image Segmentation: A Surveycited by 21
  31. 2024The Evolution of Multimodal Model Architecturescited by 21
  32. 2024Unlocking the Future: Exploring Look-Ahead Planning Mechanistic Interpretability in Large Language Modelscited by 21
  33. 2025Frontier AI's Impact on the Cybersecurity Landscapecited by 21
  34. 2023SSTFormer: Bridging Spiking Neural Network and Memory Support Transformer for Frame-Event based Recognitioncited by 20
  35. 2023Bias and Fairness in Chatbots: An Overviewcited by 20
  36. 2024Mapping the Ethics of Generative AI: A Comprehensive Scoping Reviewcited by 19
  37. 2024Leopard: A Vision Language Model For Text-Rich Multi-Image Taskscited by 18
  38. 2023Vision-Language Instruction Tuning: A Review and Analysiscited by 17

50 most cited of 153 All 153 (JSON)