mlsanityData to 7 Oct 2026

Multimodal Deep Learning

arXiv 2301.04856 ↗First posted 12 Jan 2023cs.CL cs.LG stat.MLcited by 257

Not on the map. Why?

Abstract (arXiv)

This book is the result of a seminar in which we reviewed multimodal approaches and attempted to create a solid overview of the field, starting with the current state-of-the-art approaches in the two subfields of Deep Learning individually. Further, modeling frameworks are discussed where one modality is transformed into the other, as well as models in which one modality is utilized to enhance representation learning for the other. To conclude the second part, architectures with a focus on handling both modalities simultaneously are introduced. Finally, we also cover other modalities as well as general-purpose multi-modal models, which are able to handle different tasks on different modalities within one unified architecture. One interesting application (Generative Art) eventually caps off this booklet.

Key facts. Cited by 257 papers in the map (data through 7 Oct 2026). Builds on 0 of its 31 references that are in the map. First posted on arXiv 12 Jan 2023.

Builds on

0 of 31 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 57; 2024: 62; 2025: 89; 2026: 49.

257 papers.

Sort:

By area:

  1. 2023Towards Generalist Biomedical AIcited by 286
  2. 2023Multimodal Large Language Models: A Surveycited by 193
  3. 2023Machine Unlearning: Solutions and Challengescited by 85
  4. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 74
  5. 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
  6. 2024FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusioncited by 67
  7. 2023Identifiability Results for Multimodal Contrastive Learningcited by 58
  8. 2023On Uni-Modal Feature Learning in Supervised Multi-Modal Learningcited by 53
  9. 2024UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them Allcited by 49
  10. 2023Understanding Multimodal Contrastive Learning and Incorporating Unpaired Datacited by 43
  11. 2023Enhancing multimodal cooperation via sample-level modality valuationcited by 43
  12. 2023Chat with the Environment: Interactive Multimodal Perception Using Large Language Modelscited by 42
38 more
  1. 2024What to align in multimodal contrastive learning?cited by 41
  2. 2023A Light Weight Model for Active Speaker Detectioncited by 39
  3. 2023Efficient Multimodal Fusion via Interactive Promptingcited by 32
  4. 2024Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectivescited by 30
  5. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  6. 2023Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applicationscited by 27
  7. 2023Multi-modal Machine Learning in Engineering Design: A Review and Future Directionscited by 27
  8. 2023Understanding Transferable Representation Learning and Zero-shot Transfer in CLIPcited by 26
  9. 2023A Theory of Multimodal Learningcited by 25
  10. 2025A Closer Look at Multimodal Representation Collapsecited by 25
  11. 2024From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generationcited by 23
  12. 2024OmniBind: Teach to Build Unequal-Scale Modality Interaction for Omni-Bind of Allcited by 20
  13. 2025Principled Multimodal Representation Learningcited by 19
  14. 2023UniCat: Crafting a Stronger Fusion Baseline for Multimodal Re-Identificationcited by 18
  15. 2023FedMEKT: Distillation-based Embedding Knowledge Transfer for Multimodal Federated Learningcited by 17
  16. 2025Graph4MM: Weaving Multimodal Learning with Structural Informationcited by 17
  17. 2023Recent Advances and Applications of Machine Learning in Experimental Solid Mechanics: A Reviewcited by 15
  18. 2024Distilling Privileged Multimodal Information for Expression Recognition using Optimal Transportcited by 15
  19. 2024MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AIcited by 15
  20. 2023Exploiting the Complementarity of 2D and 3D Networks to Address Domain-Shift in 3D Semantic Segmentationcited by 14
  21. 2023UniS-MMC: Multimodal Classification via Unimodality-supervised Multimodal Contrastive Learningcited by 13
  22. 2023CLIP-QDA: An Explainable Concept Bottleneck Modelcited by 13
  23. 2024Joint Multimodal Transformer for Emotion Recognition in the Wildcited by 13
  24. 2023FRAMU: Attention-based Machine Unlearning using Federated Reinforcement Learningcited by 12
  25. 2023CARAT: Contrastive Feature Reconstruction and Aggregation for Multi-Modal Multi-Label Emotion Recognitioncited by 12
  26. 2024A Survey on Integrated Sensing, Communication, and Computationcited by 12
  27. 2024Recent Advances in Multimodal Affective Computing: An NLP Perspectivecited by 12
  28. 2025Rethinking Multimodal Learning from the Perspective of Mitigating Classification Ability Disproportioncited by 12
  29. 2025Closing the Modality Gap for Mixed Modality Searchcited by 11
  30. 2023Speaker Recognition in Realistic Scenario Using Multimodal Datacited by 9
  31. 2023Multimodal Fusion Interactions: A Study of Human and Automatic Quantificationcited by 9
  32. 2025Knowledge Bridger: Towards Training-free Missing Modality Completioncited by 9
  33. 2023Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videoscited by 8
  34. 2024NeuroNURBS: Learning Efficient Surface Representations for 3D Solidscited by 8
  35. 2025Improving Multimodal Learning Balance and Sufficiency through Data Remixingcited by 8
  36. 2023Performance Optimization using Multimodal Modeling and Heterogeneous GNNcited by 7
  37. 2024M2CURL: Sample-Efficient Multimodal Reinforcement Learning via Self-Supervised Representation Learning for Robotic Manipulationcited by 7
  38. 2024Multimodal Object Detection using Depth and Image Data for Manufacturing Partscited by 7

50 most cited of 257 All 257 (JSON)