mlsanityData to 7 Oct 2026

A Review on Methods and Applications in Multimodal Deep Learning

arXiv 2202.09195 ↗First posted 18 Feb 2022cs.LG cs.MMcited by 29

Not on the map. Why?

Abstract (arXiv)

Deep Learning has implemented a wide range of applications and has become increasingly popular in recent years. The goal of multimodal deep learning (MMDL) is to create models that can process and link information using various modalities. Despite the extensive development made for unimodal learning, it still cannot cover all the aspects of human learning. Multimodal learning helps to understand and analyze better when various senses are engaged in the processing of information. This paper focuses on multiple types of modalities, i.e., image, video, text, audio, body gestures, facial expressions, and physiological signals. Detailed analysis of the baseline approaches and an in-depth study of recent advancements during the last five years (2017 to 2021) in multimodal deep learning applications has been provided. A fine-grained taxonomy of various multimodal deep learning methods is proposed, elaborating on different applications in more depth. Lastly, main issues are highlighted separately for each domain, along with their possible future research directions.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 72 of its 180 references that are in the map. First posted on arXiv 18 Feb 2022.

Builds on

72 of 180 references.

References · 72

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  3. 2015VQA: Visual Question Answeringcited by 2,489
  4. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  5. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  6. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  7. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  8. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  9. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  10. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  11. 2017Dense-Captioning Events in Videoscited by 789
  12. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
38 more
  1. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  2. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  3. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  4. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  5. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  6. 2020Experience Grounds Languagecited by 241
  7. 2019EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognitioncited by 226
  8. 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 214
  9. 2018Talking Face Generation by Adversarially Disentangled Audio-Visual Representationcited by 212
  10. 2020RSVQA: Visual Question Answering for Remote Sensing Datacited by 197
  11. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  12. 2015Exploring Models and Data for Image Question Answeringcited by 139
  13. 2017MUTAN: Multimodal Tucker Fusion for Visual Question Answeringcited by 137
  14. 2020More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classificationcited by 126
  15. 2016Visual Question Answering: A Survey of Methods and Datasetscited by 123
  16. 2019Relation-Aware Graph Attention Network for Visual Question Answeringcited by 119
  17. 2017Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answeringcited by 117
  18. 2014A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Inputcited by 112
  19. 2016A Survey of Multi-View Representation Learningcited by 107
  20. 2018Neural Voice Cloning with a Few Samplescited by 100
  21. 2017Visual to Sound: Generating Natural Sound for Videos in the Wildcited by 98
  22. 2018Reconstruction Network for Video Captioningcited by 98
  23. 2017An Analysis of Visual Question Answering Algorithmscited by 90
  24. 2019Multimodal Intelligence: Representation Learning, Information Fusion, and Applicationscited by 89
  25. 2017Deep Voice: Real-time Neural Text-to-Speechcited by 83
  26. 2018Unsupervised Image Captioningcited by 79
  27. 2017Deep Voice 2: Multi-Speaker Neural Text-to-Speechcited by 75
  28. 2019MUREL: Multimodal Relational Reasoning for Visual Question Answeringcited by 68
  29. 2019Memory-Attended Recurrent Network for Video Captioningcited by 65
  30. 20182.5D Visual Soundcited by 63
  31. 2019Spatio-Temporal Dynamics and Semantic Attribute Enriched Visual Encoding for Video Captioningcited by 60
  32. 2019Streamlined Dense Video Captioningcited by 57
  33. 2019M3ER: Multiplicative Multimodal Emotion Recognition Using Facial, Textual, and Speech Cuescited by 56
  34. 2019Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methodscited by 55
  35. 2018Less Is More: Picking Informative Frames for Video Captioningcited by 52
  36. 2020Parallel Tacotron: Non-Autoregressive and Controllable TTScited by 49
  37. 2015SentiCap: Generating Image Descriptions with Sentimentscited by 48
  38. 2020Improving Image Captioning by Leveraging Intra- and Inter-layer Global Representation in Transformer Networkcited by 48

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 10; 2025: 9; 2026: 3.

29 papers.

Sort:

By area:

  1. 2023Multimodal Large Language Models: A Surveycited by 193
  2. 2025MLLMs are Deeply Affected by Modality Biascited by 38
  3. 2024A Systematic Review of Intermediate Fusion in Multimodal Deep Learning for Biomedical Applicationscited by 25
  4. 2025Graph4MM: Weaving Multimodal Learning with Structural Informationcited by 17
  5. 2024Enhancing Visual Question Answering through Ranking-Based Hybrid Training and Multimodal Fusioncited by 16
  6. 2023Revisiting Disentanglement and Fusion on Modality and Context in Conversational Multimodal Emotion Recognitioncited by 13
  7. 2024RankCLIP: Ranking-Consistent Language-Image Pretrainingcited by 9
  8. 2024Robustness of Structured Data Extraction from In-plane Rotated Documents using Multi-Modal Large Language Models (LLM)cited by 7
  9. 2025A Deep Learning Framework for Sequence Mining with Bidirectional LSTM and Multi-Scale Attentioncited by 7
  10. 2024DisCoM-KD: Cross-Modal Knowledge Distillation via Disentanglement Representation and Adversarial Learningcited by 6
  11. 2023VulnSense: Efficient Vulnerability Detection in Ethereum Smart Contracts by Multimodal Learning with Graph Neural Network and Language Modelcited by 4
  12. 2023A Survey on Image-text Multimodal Modelscited by 4
17 more
  1. 2024Multimodal Deep Learning for Low-Resource Settings: A Vector Embedding Alignment Approach for Healthcare Applicationscited by 4
  2. 2023Cascaded Cross-Modal Transformer for Request and Complaint Detectioncited by 3
  3. 2025Multimodal Representation Learning and Fusioncited by 3
  4. 2023Integrating spoken instructions into flight trajectory prediction to optimize automation in air traffic controlcited by 2
  5. 2024Cascaded Cross-Modal Transformer for Audio-Textual Classificationcited by 2
  6. 2024Intelligent Cross-Organizational Process Mining: A Survey and New Perspectivescited by 2
  7. 2025An evaluation of LLMs for generating movie reviews: GPT-4o, Gemini-2.0 and DeepSeek-V3cited by 2
  8. 2024Stitching Gaps: Fusing Situated Perceptual Knowledge with Vision Transformers for High-Level Image Classificationcited by 1
  9. 2025TabulaTime: A Novel Multimodal Deep Learning Framework for Advancing Acute Coronary Syndrome Prediction through Environmental and Clinical Data Integrationcited by 1
  10. 2025Multi-Modal Continual Learning via Cross-Modality Adapters and Representation Alignment with Knowledge Preservationcited by 1
  11. 2023Content-based jewellery item retrieval using the local region-based histogramscited by 0
  12. 2024Integrative Variational Autoencoders for Generative Modeling of an Image Outcome with Multiple Input Imagescited by 0
  13. 2025Bimodal Connection Attention Fusion for Speech Emotion Recognitioncited by 0
  14. 2025Advancing Compositional LLM Reasoning with Structured Task Relations in Interactive Multimodal Communicationscited by 0
  15. 2026CSI-tuples-based 3D Channel Fingerprints Construction Assisted by MultiModal Learningcited by 0
  16. 2026Multimodal Domain Generalization for Depression Detection: An Attention-Based BiLSTM Network with Domain-Adversarial Trainingcited by 0
  17. 2026Deep Multimodal Fusion Detection through Spatial Mask and Channel Competitioncited by 0