mlsanityData to 14 Oct 2026

Multimodal Machine Learning: A Survey and Taxonomy

arXiv 1705.09406 ↗First posted 26 May 2017cs.LGcited by 1,013

Not on the map. Why?

Cited by 34 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Our experience of the world is multimodal - we see objects, hear sounds, feel texture, smell odors, and taste flavors. Modality refers to the way in which something happens or is experienced and a research problem is characterized as multimodal when it includes multiple such modalities. In order for Artificial Intelligence to make progress in understanding the world around us, it needs to be able to interpret such multimodal signals together. Multimodal machine learning aims to build models that can process and relate information from multiple modalities. It is a vibrant multi-disciplinary field of increasing importance and with extraordinary potential. Instead of focusing on specific multimodal applications, this paper surveys the recent advances in multimodal machine learning itself and presents them in a common taxonomy. We go beyond the typical early and late fusion categorization and identify broader challenges that are faced by multimodal machine learning, namely: representation, translation, alignment, fusion, and co-learning. This new taxonomy will enable researchers to better understand the state of the field and identify directions for future research.

Key facts. Cited by 1,013 papers in the map (data through 14 Oct 2026). First posted on arXiv 26 May 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 113; 2022: 155; 2023: 182; 2024: 160; 2025: 213; 2026: 190.

1,013 papers.

Sort:

By area:

  1. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  2. 2021Deep Neural Networks and Tabular Data: A Surveycited by 491
  3. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  4. 2022Multimodal Learning with Transformers: A Surveycited by 400
  5. 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Modelcited by 199
  6. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  7. 2021Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysiscited by 180
  8. 2021Synthesis of Compositional Animations from Textual Descriptionscited by 178
  9. 2023Towards Open Vocabulary Learning: A Surveycited by 166
  10. 2021What Makes Multi-modal Learning Better than Single (Provably)cited by 153
  11. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 146
  12. 2022Contrastive Audio-Visual Masked Autoencodercited by 146
38 more
  1. 2022X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrievalcited by 128
  2. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  3. 2022Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)cited by 122
  4. 2021Data Collection and Quality Challenges in Deep Learning: A Data-Centric AI Perspectivecited by 116
  5. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  6. 2022A Survey of Multi-Agent Deep Reinforcement Learning with Communicationcited by 104
  7. 2023Inter-X: Towards Versatile Human-Human Interaction Analysiscited by 98
  8. 2022Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networkscited by 95
  9. 2021Adversarial Sticker: A Stealthy Attack Method in the Physical Worldcited by 93
  10. 2024Deep Multimodal Learning with Missing Modality: A Surveycited by 92
  11. 2021Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Predictioncited by 90
  12. 2021Generalized Multimodal ELBOcited by 87
  13. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 85
  14. 2023A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?cited by 84
  15. 2023Quantifying & Modeling Multimodal Interactions: An Information Decomposition Frameworkcited by 83
  16. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 83
  17. 2023Provable Dynamic Fusion for Low-Quality Multimodal Datacited by 82
  18. 2022Knowledge Tracing: A Surveycited by 77
  19. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 76
  20. 2022Multi-Modal Knowledge Graph Construction and Application: A Surveycited by 74
  21. 2024Multimodal Alignment and Fusion: A Surveycited by 74
  22. 2021TorchGeo: Deep Learning With Geospatial Datacited by 73
  23. 2023Boosting Multi-modal Model Performance with Adaptive Gradient Modulationcited by 73
  24. 2024MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistancecited by 73
  25. 2022M3Care: Learning with Missing Modalities in Multimodal Healthcare Datacited by 72
  26. 2023ALIP: Adaptive Language-Image Pre-training with Synthetic Captioncited by 70
  27. 2021Including Signed Languages in Natural Language Processingcited by 68
  28. 2023Grounding 3D Object Affordance from 2D Interactions in Imagescited by 62
  29. 2022UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognitioncited by 61
  30. 2023Identifiability Results for Multimodal Contrastive Learningcited by 61
  31. 2024ReconBoost: Boosting Can Achieve Modality Reconcilementcited by 60
  32. 2024AVFF: Audio-Visual Feature Fusion for Video Deepfake Detectioncited by 59
  33. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  34. 2021Reconsidering Representation Alignment for Multi-view Clusteringcited by 57
  35. 2023Deception Detection from Linguistic and Physiological Data Streams Using Bimodal Convolutional Neural Networkscited by 56
  36. 2022Image-text Retrieval: A Survey on Recent Research and Developmentcited by 55
  37. 2022Dynamic Multimodal Fusioncited by 55
  38. 2022IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languagescited by 54

50 most cited of 1,013 All 1,013 (JSON)