mlsanityData to 14 Oct 2026

Multimodal Learning with Transformers: A Survey

arXiv 2206.06488 ↗First posted 13 Jun 2022cs.CV cs.LGcited by 400

Not on the map. Why?

Cited by 23 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformer is a promising neural network learner, and has achieved great success in various machine learning tasks. Thanks to the recent prevalence of multimodal applications and big data, Transformer-based multimodal learning has become a hot topic in AI research. This paper presents a comprehensive survey of Transformer techniques oriented at multimodal data. The main contents of this survey include: (1) a background of multimodal learning, Transformer ecosystem, and the multimodal big data era, (2) a theoretical review of Vanilla Transformer, Vision Transformer, and multimodal Transformers, from a geometrically topological perspective, (3) a review of multimodal Transformer applications, via two important paradigms, i.e., for multimodal pretraining and for specific multimodal tasks, (4) a summary of the common challenges and designs shared by the multimodal Transformer models and applications, and (5) a discussion of open problems and potential research directions for the community.

Key facts. Cited by 400 papers in the map (data through 14 Oct 2026). Builds on 285 of its 358 references that are in the map. First posted on arXiv 13 Jun 2022.

Builds on

285 of 358 references.

References · 285

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  9. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  10. 2020End-to-End Object Detection with Transformerscited by 7,414
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  12. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
38 more
  1. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  3. 2016Layer Normalizationcited by 5,669
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  8. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  11. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  12. 2020Longformer: The Long-Document Transformercited by 3,413
  13. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  14. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  15. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  16. 2015VQA: Visual Question Answeringcited by 2,506
  17. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  18. 2017Non-local Neural Networkscited by 2,293
  19. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  20. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  21. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  22. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  23. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  24. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  25. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  26. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  27. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  28. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  29. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  30. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  31. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  32. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,167
  33. 2021Grounded Language-Image Pre-trainingcited by 1,156
  34. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
  35. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  36. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
  37. 2021Transformers in Vision: A Surveycited by 1,013
  38. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954

50 most cited of 285 All 285 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 10; 2023: 86; 2024: 126; 2025: 111; 2026: 67.

400 papers.

Sort:

By area:

  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  2. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 210
  3. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 194
  4. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 169
  5. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  6. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  7. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  8. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 110
  9. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 110
  10. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 108
  11. 2022A Survey of Knowledge Graph Reasoning on Graph Types: Static, Dynamic, and Multimodalcited by 106
  12. 2023Inter-X: Towards Versatile Human-Human Interaction Analysiscited by 98
38 more
  1. 2023Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Predictioncited by 92
  2. 2023A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?cited by 84
  3. 2023Quantifying & Modeling Multimodal Interactions: An Information Decomposition Frameworkcited by 83
  4. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 83
  5. 2023A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4cited by 81
  6. 2024A Survey on Visual Anomaly Detection: Challenge, Approach, and Prospectcited by 69
  7. 2024TR-DETR: Task-Reciprocal Transformer for Joint Moment Retrieval and Highlight Detectioncited by 59
  8. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  9. 2022Multimodal learning with graphscited by 54
  10. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  11. 2024Toward Visual Grounding: A Surveycited by 52
  12. 2024Multimodal Prototyping for cancer survival predictioncited by 47
  13. 2024What to align in multimodal contrastive learning?cited by 47
  14. 2024Towards a Foundation Model for Partial Differential Equations: Multi-Operator Learning and Extrapolationcited by 46
  15. 2023Can We Edit Multimodal Large Language Models?cited by 44
  16. 2023OmniVec: Learning robust representations with cross modal sharingcited by 44
  17. 2023Understanding Multimodal Contrastive Learning and Incorporating Unpaired Datacited by 43
  18. 2024Diagnosing and Re-learning for Balanced Multimodal Learningcited by 41
  19. 2023MH-DETR: Video Moment and Highlight Detection with Cross-modal Transformercited by 40
  20. 2025MLLMs are Deeply Affected by Modality Biascited by 40
  21. 2023Transformers in Speech Processing: A Surveycited by 39
  22. 2024Recent Advances of Multimodal Continual Learning: A Comprehensive Surveycited by 39
  23. 2024PROSE-FD: A Multimodal PDE Foundation Model for Learning Multiple Operators for Forecasting Fluid Dynamicscited by 34
  24. 2023Audio-aware Query-enhanced Transformer for Audio-Visual Segmentationcited by 33
  25. 2025Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMscited by 31
  26. 2024What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluationcited by 30
  27. 2023CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservationcited by 29
  28. 2023Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptationcited by 29
  29. 2024Mosaic of Modalities: A Comprehensive Benchmark for Multimodal Graph Learningcited by 29
  30. 2023Learning to Ground Instructional Articles in Videos through Narrationscited by 28
  31. 2023RoadFormer: Duplex Transformer for RGB-Normal Semantic Road Scene Parsingcited by 27
  32. 2024DLF: Disentangled-Language-Focused Multimodal Sentiment Analysiscited by 27
  33. 2023RPEFlow: Multimodal Fusion of RGB-PointCloud-Event for Joint Optical Flow and Scene Flow Estimationcited by 26
  34. 2024A Systematic Review of Intermediate Fusion in Multimodal Deep Learning for Biomedical Applicationscited by 25
  35. 2022CKD-TransBTS: Clinical Knowledge-Driven Hybrid Transformer with Modality-Correlated Cross-Attention for Brain Tumor Segmentationcited by 24
  36. 2024Materials science in the era of large language models: a perspectivecited by 24
  37. 2025DecAlign: Hierarchical Cross-Modal Alignment for Decoupled Multimodal Representation Learningcited by 24
  38. 2023Robust Visual Question Answering: Datasets, Methods, and Future Challengescited by 23

50 most cited of 400 All 400 (JSON)