mlsanityData to 7 Oct 2026

Multimodal Alignment and Fusion: A Survey

arXiv 2411.17040 ↗First posted 26 Nov 2024cs.CVcited by 73

Not on the map. Why?

Abstract (arXiv)

This survey provides a comprehensive overview of recent advances in multimodal alignment and fusion within the field of machine learning, driven by the increasing availability and diversity of data modalities such as text, images, audio, and video. Unlike previous surveys that often focus on specific modalities or limited fusion strategies, our work presents a structure-centric and method-driven framework that emphasizes generalizable techniques. We systematically categorize and analyze key approaches to alignment and fusion through both structural perspectives -- data-level, feature-level, and output-level fusion -- and methodological paradigms -- including statistical, kernel-based, graphical, generative, contrastive, attention-based, and large language model (LLM)-based methods, drawing insights from an extensive review of over 260 relevant studies. Furthermore, this survey highlights critical challenges such as cross-modal misalignment, computational bottlenecks, data quality issues, and the modality gap, along with recent efforts to address them. Applications ranging from social media analysis and medical imaging to emotion recognition and embodied AI are explored to illustrate the real-world impact of robust multimodal systems. The insights provided aim to guide future research toward optimizing multimodal learning systems for improved scalability, robustness, and generalizability across diverse domains.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Builds on 157 of its 271 references that are in the map. First posted on arXiv 26 Nov 2024.

Builds on

157 of 271 references.

References · 157

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2024The Llama 3 Herd of Modelscited by 14,192
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  12. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
38 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  9. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  10. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  11. 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentationcited by 1,483
  12. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  13. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  14. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  15. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  16. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  17. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  18. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  19. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  20. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  21. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  22. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  23. 2023A Survey on Multimodal Large Language Modelscited by 898
  24. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  25. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  26. 2024Yi: Open Foundation Models by 01.AIcited by 706
  27. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  28. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  29. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  30. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  31. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  32. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  33. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  34. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
  35. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  36. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  37. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  38. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536

50 most cited of 157 All 157 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 42; 2026: 31.

73 papers.

Sort:

By area:

  1. 2025Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawingcited by 130
  2. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  3. 2025Large Language Models for Multi-Robot Systems: A Surveycited by 26
  4. 2025Geometrically-Constrained Agent for Spatial Reasoningcited by 25
  5. 2025MathAgent: Leveraging a Mixture-of-Math-Agent Framework for Real-World Multimodal Mathematical Error Detectioncited by 24
  6. 2025Lost in Embeddings: Information Loss in Vision-Language Modelscited by 18
  7. 2025Multimodal Emotion Recognition in Conversations: A Survey of Methods, Trends, Challenges and Prospectscited by 10
  8. 2026Towards Long-horizon Agentic Multimodal Searchcited by 9
  9. 2025Joint Fusion and Encoding: Advancing Multimodal Retrieval from the Ground Upcited by 6
  10. 2025VDRive: Leveraging Reinforced VLA and Diffusion Policy for End-to-end Autonomous Drivingcited by 5
  11. 2025A Review of 3D Object Detection with Vision-Language Modelscited by 4
  12. 2025M2BeamLLM: Multimodal Sensing-empowered mmWave Beam Prediction with Large Language Modelscited by 4
38 more
  1. 2025BabyVLM-V2: Toward Developmentally Grounded Pretraining and Benchmarking of Vision Foundation Modelscited by 4
  2. 2025Embodied AI with Foundation Models for Mobile Service Robots: A Systematic Reviewcited by 3
  3. 2025Multimodal Representation Learning and Fusioncited by 3
  4. 2025Enhancing 3D Semantic Scene Completion with a Refinement Modulecited by 3
  5. 2025Integrating Biological and Machine Intelligence: Attention Mechanisms in Brain-Computer Interfacescited by 2
  6. 2025Towards LLM-Centric Multimodal Fusion: A Survey on Integration Strategies and Techniquescited by 2
  7. 2025Training-Free Multimodal Large Language Model Orchestrationcited by 2
  8. 2025Multimodal Data Storage and Retrieval for Embodied AI: A Surveycited by 2
  9. 2026The Semantic Lifecycle in Embodied AI: Acquisition, Representation and Storage via Foundation Modelscited by 2
  10. 2026HERMES: A Unified Open-Source Framework for Realtime Multimodal Physiological Sensing, Edge AI, and Intervention in Closed-Loop Smart Healthcare Applicationscited by 2
  11. 2026When Understanding Becomes a Risk: Authenticity and Safety Risks in the Emerging Image Generation Paradigmcited by 2
  12. 2026Tactile-based Multimodal Fusion in Embodied Intelligence: A Survey of Vision, Language, and Contact-Driven Paradigmscited by 2
  13. 2025Semantic-Space-Intervened Diffusive Alignment for Visual Classificationcited by 1
  14. 2025Robotic Perception with a Large Tactile-Vision-Language Model for Physical Property Inferencecited by 1
  15. 2025SlideMamba: Entropy-Based Adaptive Fusion of GNN and Mamba for Enhanced Representation Learning in Digital Pathologycited by 1
  16. 2025From Classification to Cross-Modal Understanding: Leveraging Vision-Language Models for Fine-Grained Renal Pathologycited by 1
  17. 2025BotaCLIP: Contrastive Learning for Botany-Aware Representation of Earth Observation Datacited by 1
  18. 2025Explainable Multimodal Regression via Information Decompositioncited by 1
  19. 2025RAH-VLA: Resolution-Adaptive Hierarchical Vision-Language Alignment for Multimodal Remote Sensing Understandingcited by 1
  20. 2026MEIDNet: Multimodal generative AI framework for inverse materials designcited by 1
  21. 2026Chatting with Images for Introspective Visual Thinkingcited by 1
  22. 2026Memory Bear AI Memory Science Engine for Multimodal Affective Intelligence: A Technical Reportcited by 1
  23. 2026Contrastive Learning Boosts Deterministic and Generative Models for Weather Datacited by 1
  24. 2026Multi-Perspective Evidence Synthesis and Reasoning for Unsupervised Multimodal Entity Linkingcited by 1
  25. 2026VTBench: A Multimodal Framework for Time-Series Classification with Chart-Based Representationscited by 1
  26. 2026PaperFit: Vision-in-the-Loop Typesetting Optimization for Scientific Documentscited by 1
  27. 2026Artificial Intelligence for the Characterization of Particles and Fibers by Optical Microscopycited by 1
  28. 2025Towards Achieving Perfect Multimodal Alignmentcited by 0
  29. 2025Capturing Fine-Grained Alignments Improves 3D Affordance Detectioncited by 0
  30. 2025Explainability Through Systematicity: The Hard Systematicity Challenge for Artificial Intelligencecited by 0
  31. 2025From Detection to Correction: Backdoor-Resilient Face Recognition via Vision-Language Trigger Detection and Noise-Based Neutralizationcited by 0
  32. 2025MUJICA: Reforming SISR Models for PBR Material Super-Resolution via Cross-Map Attentioncited by 0
  33. 2025A Systematic Review on the Generative AI Applications in Human Medical Genomicscited by 0
  34. 2025Integrated Multivariate Segmentation Tree for Heterogeneous Credit Data Analysis in Small- and Medium-Sized Enterprisescited by 0
  35. 2025MVRS: The Multimodal Virtual Reality Stimuli-based Emotion Recognition Datasetcited by 0
  36. 2025InstructPLM-mu: 1-Hour Fine-Tuning of ESM2 Beats ESM3 in Protein Mutation Predictionscited by 0
  37. 2025MultiFair: Multimodal Balanced Fairness-Aware Medical Classification with Dual-Level Gradient Modulationcited by 0
  38. 2025Efficient Generalization via Multimodal Co-Training under Data Scarcity and Distribution Shiftcited by 0

50 most cited of 73 All 73 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024AlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-modal Alignment
  2. 2025Aligning Multimodal LLM with Human Preference: A Survey
  3. 2026Explicit Representation Alignment for Multimodal Sentiment Analysis
  4. 2023How to Bridge the Gap between Modalities: Survey on Multimodal Large Language Model
  5. 2026AlignMamba-2: Enhancing Multimodal Fusion and Sentiment Analysis with Modality-Aware Mamba
  6. 2024Understanding Alignment in Multimodal LLMs: A Comprehensive Study
  7. 2025SAE-V: Interpreting Multimodal Models for Enhanced Alignment
  8. 2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs