mlsanityData to 7 Oct 2026

From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalities

arXiv 2412.11694 ↗First posted 16 Dec 2024cs.AI cs.CL cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

To tackle complex tasks in real-world scenarios, more researchers are focusing on Omni-MLLMs, which aim to achieve omni-modal understanding and generation. Beyond the constraints of any specific non-linguistic modality, Omni-MLLMs map various non-linguistic modalities into the embedding space of LLMs and enable the interaction and understanding of arbitrary combinations of modalities within a single model. In this paper, we systematically investigate relevant research and provide a comprehensive survey of Omni-MLLMs. Specifically, we first explain the four core components of Omni-MLLMs for unified multi-modal modeling with a meticulous taxonomy that offers novel perspectives. Then, we introduce the effective integration achieved through two-stage training and discuss the corresponding datasets as well as evaluation. Furthermore, we summarize the main challenges of current Omni-MLLMs and outline future directions. We hope this paper serves as an introduction for beginners and promotes the advancement of related research. Resources have been made publicly available at https://github.com/threegold116/Awesome-Omni-MLLMs.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 205 of its 297 references that are in the map. First posted on arXiv 16 Dec 2024.

Builds on

205 of 297 references.

References · 205

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  12. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
38 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  2. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  3. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  10. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  12. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  13. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  14. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  15. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  16. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  17. 2021Vision Transformers for Dense Predictioncited by 1,789
  18. 2019Towards VQA Models That Can Readcited by 1,696
  19. 2021ViViT: A Video Vision Transformercited by 1,610
  20. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  21. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  22. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  23. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  24. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  25. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  26. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  27. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  28. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  29. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  30. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  31. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  32. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  33. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  34. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  35. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  36. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  37. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  38. 2016Modeling Context in Referring Expressionscited by 1,054

50 most cited of 205 All 205 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 17.

22 papers.

Sort:

By area:

  1. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 27
  2. 2026OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-modal Large Language Modelscited by 23
  3. 2025Active Perception Agent for Omnimodal Audio-Video Understandingcited by 13
  4. 2025OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Mergingcited by 12
  5. 2026SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Modelscited by 12
  6. 2026OmniRAG-Agent: Agentic Omnimodal Reasoning for Low-Resource Long Audio-Video Question Answeringcited by 5
  7. 2026Omni-Safety under Cross-Modality Conflict: Vulnerabilities, Dynamics Mechanisms and Efficient Alignmentcited by 4
  8. 2026From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agentscited by 4
  9. 2026SONIC-O1: A Real-World Benchmark for Evaluating Multimodal Large Language Models on Audio-Video Understandingcited by 3
  10. 2025Towards LLM-Centric Multimodal Fusion: A Survey on Integration Strategies and Techniquescited by 2
  11. 2026Graph-to-Frame RAG: Visual-Space Knowledge Fusion for Training-Free and Auditable Video Reasoningcited by 2
  12. 2026Beyond Text-Dominance: Understanding Modality Preference of Omni-modal Large Language Modelscited by 2
10 more
  1. 2026OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Modelscited by 2
  2. 2025Towards Multimodal Graph Large Language Modelcited by 0
  3. 2026MAPLE: Modality-Aware Post-training and Learning Ecosystemcited by 0
  4. 2026IncompeBench: A Permissively Licensed, Fine-Grained Benchmark for Music Information Retrievalcited by 0
  5. 2026AGCD: Agent-Guided Cross-Modal Decoding for Weather Forecastingcited by 0
  6. 2026Demographic and Linguistic Bias Evaluation in Omnimodal Language Modelscited by 0
  7. 2026ArtiFact: A Large-Scale Multi-Modal Cultural Heritage Datasetcited by 0
  8. 2026Evaluating Multimodal LLMs across Text and Audio Modalities for Accessible Disaster Assistancecited by 0
  9. 2026Tri-PvP: Exposing Modality Bias in Omni-Modal Large Language Models through Perceptual-Propositional Evidence Conflictscited by 0
  10. 2026OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialoguecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Capybara-OMNI: An Efficient Paradigm for Building Omni-Modal Language Models
  2. 2023How to Bridge the Gap between Modalities: Survey on Multimodal Large Language Model
  3. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective
  4. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performance
  5. 2024OmniBench: Towards The Future of Universal Omni-Language Models
  6. 2025Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs
  7. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  8. 2024Explore the Limits of Omni-modal Pretraining at Scale