mlsanityData to 7 Oct 2026

Towards LLM-Centric Multimodal Fusion: A Survey on Integration Strategies and Techniques

arXiv 2506.04788 ↗First posted 5 Jun 2025cs.CL cs.AI cs.LGcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The rapid progress of Multimodal Large Language Models(MLLMs) has transformed the AI landscape. These models combine pre-trained LLMs with various modality encoders. This integration requires a systematic understanding of how different modalities connect to the language backbone. Our survey presents an LLM-centric analysis of current approaches. We examine methods for transforming and aligning diverse modal inputs into the language embedding space. This addresses a significant gap in existing literature. We propose a classification framework for MLLMs based on three key dimensions. First, we examine architectural strategies for modality integration. This includes both the specific integration mechanisms and the fusion level. Second, we categorize representation learning techniques as either joint or coordinate representations. Third, we analyze training paradigms, including training strategies and objective functions. By examining 125 MLLMs developed between 2021 and 2025, we identify emerging patterns in the field. Our taxonomy provides researchers with a structured overview of current integration techniques. These insights aim to guide the development of more robust multimodal integration strategies for future models built on pre-trained foundations.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 65 of its 96 references that are in the map. First posted on arXiv 5 Jun 2025.

Builds on

65 of 96 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  2. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks12 Jun
  3. 2024Efficient Multimodal Large Language Models: A Survey17 May
  4. 2024Auto-Encoding Morph-Tokens for Multimodal LLM3 May
  5. 2024The Revolution of Multimodal Large Language Models: A Survey19 Feb
  6. 2023VILA: On Pre-training for Visual Language Models12 Dec
  7. 2023Honeybee: Locality-enhanced Projector for Multimodal LLM11 Dec
  8. 2023Hijacking Context in Large Multi-modal Models7 Dec
  9. 2023Multimodal Large Language Models: A Survey22 Nov
  10. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
  11. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  12. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities18 May
1 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  9. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  10. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  11. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  12. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
38 more
  1. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  2. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  3. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  4. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  5. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  6. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  7. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  8. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  9. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  10. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  11. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 344
  12. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  13. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  14. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  15. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  16. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 260
  17. 2023Multimodal Deep Learningcited by 257
  18. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  19. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  20. 2023Pengi: An Audio Language Model for Audio Taskscited by 217
  21. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
  22. 2023RSGPT: A Remote Sensing Vision Language Model and Benchmarkcited by 204
  23. 2023ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulationcited by 157
  24. 2023MotionGPT: Finetuned LLMs Are General-Purpose Motion Generatorscited by 145
  25. 2024LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Modelcited by 133
  26. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  27. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Executioncited by 116
  28. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 100
  29. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 79
  30. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 78
  31. 2024SNIFFER: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detectioncited by 73
  32. 2024Multimodal Alignment and Fusion: A Surveycited by 73
  33. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
  34. 2024Image Fusion via Vision-Language Modelcited by 40
  35. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 37
  36. 2024Can't make an Omelette without Breaking some Eggs: Plausible Action Anticipation using Large Video-Language Modelscited by 32
  37. 2024Enhancing Visual Document Understanding with Contrastive Learning in Large Visual-Language Modelscited by 30
  38. 2024Beyond Text: Frozen Large Language Models in Visual Signal Comprehensioncited by 23

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Robot Confirmation Generation and Action Planning Using Long-context Q-Former Integrated with Multimodal LLMcited by 0
  2. 2026Leveraging Visual Signals for Robust Token-Level Uncertainty in Vision-Language Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Model Composition for Multimodal Large Language Models
  2. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
  3. 2026Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs
  4. 2026Multimodal LLMs under Pairwise Modalities
  5. 2026Multilingual and Multimodal LLMs in the Wild: Building for Low-Resource Languages
  6. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts
  7. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
  8. 2026Sparse Shortcuts: Facilitating Efficient Fusion in Multimodal Large Language Models