mlsanityData to 7 Oct 2026

Ola: Pushing the Frontiers of Omni-Modal Language Model

arXiv 2502.04328 ↗First posted 6 Feb 2025cs.CV cs.CL cs.MMcited by 79

Not on the map. Why?

Cited by 29 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent advances in large language models, particularly following GPT-4o, have sparked increasing interest in developing omni-modal models capable of understanding more modalities. While some open-source alternatives have emerged, there is still a notable lag behind specialized single-modality models in performance. In this paper, we present Ola, an Omni-modal Language model that achieves competitive performance across image, video, and audio understanding compared to specialized counterparts, pushing the frontiers of the omni-modal language model to a large extent. We conduct a comprehensive exploration of architectural design, data curation, and training strategies essential for building a robust omni-modal model. Ola incorporates advanced visual understanding and audio recognition capabilities through several critical and effective improvements over mainstream baselines. Moreover, we rethink inter-modal relationships during omni-modal training, emphasizing cross-modal alignment with video as a central bridge, and propose a progressive training pipeline that begins with the most distinct modalities and gradually moves towards closer modality alignment. Extensive experiments demonstrate that Ola surpasses existing open omni-modal LLMs across all modalities while achieving highly competitive performance compared to state-of-the-art specialized models of similar sizes. We aim to make Ola a fully open omni-modal understanding solution to advance future research in this emerging field. Model weights, code, and data are open-sourced at https://github.com/Ola-Omni/Ola.

Key facts. Cited by 79 papers in the map (data through 7 Oct 2026). Builds on 63 of its 78 references that are in the map. First posted on arXiv 6 Feb 2025.

Builds on

63 of 78 references.

References · 63

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  8. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  10. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  11. 2024Qwen2 Technical Reportcited by 2,076
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
38 more
  1. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  3. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  4. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  5. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  6. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  7. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  8. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  9. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  10. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  11. 2016A Diagram Is Worth A Dozen Imagescited by 793
  12. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  13. 2024Yi: Open Foundation Models by 01.AIcited by 706
  14. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 644
  15. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  16. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  17. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  18. 2024Qwen2-Audio Technical Reportcited by 543
  19. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  20. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  21. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  22. 2023MusicLM: Generating Music From Textcited by 442
  23. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  24. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  25. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  26. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  27. 2019Clotho: An Audio Captioning Datasetcited by 354
  28. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  29. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
  30. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  31. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  32. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 215
  33. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 191
  34. 2024Pixtral 12Bcited by 174
  35. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 172
  36. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 169
  37. 2023AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into Onecited by 165
  38. 2024Streaming Long Video Understanding with Large Language Modelscited by 164

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 47; 2026: 32.

79 papers.

Sort:

By area:

  1. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 206
  2. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 101
  3. 2025EgoLife: Towards Egocentric Life Assistantcited by 86
  4. 2025Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoningcited by 67
  5. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMscited by 54
  6. 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 53
  7. 2025HumanOmniV2: From Understanding to Omni-Modal Reasoning with Contextcited by 50
  8. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  9. 2025ViSpeak: Visual Instruction Feedback in Streaming Videoscited by 38
  10. 2025AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestrationcited by 26
  11. 2025Streaming Video Instruction Tuningcited by 25
  12. 2025Towards Video Thinking Test: A Holistic Benchmark for Advanced Video Reasoning and Understandingcited by 24
38 more
  1. 2025LongCat-Flash-Omni Technical Reportcited by 23
  2. 2025ShotBench: Expert-Level Cinematic Understanding in Vision-Language Modelscited by 22
  3. 2025EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMscited by 22
  4. 2026LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMscited by 22
  5. 2025SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMscited by 21
  6. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performancecited by 19
  7. 2025See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Modelscited by 19
  8. 2025R-4B: Incentivizing General-Purpose Auto-Thinking Capability in MLLMs via Bi-Mode Annealing and Reinforce Learningcited by 18
  9. 2025AV-Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMscited by 15
  10. 2025SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearingcited by 14
  11. 2025MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talkscited by 14
  12. 2025High-Resolution Visual Reasoning via Multi-Turn Grounding-Based Reinforcement Learningcited by 13
  13. 2025Active Perception Agent for Omnimodal Audio-Video Understandingcited by 13
  14. 2025RefineShot: Rethinking Cinematography Understanding with Foundational Skill Evaluationcited by 10
  15. 2025Understanding the Modality Gap: An Empirical Study on the Speech-Text Alignment Mechanism of Large Speech Language Modelscited by 10
  16. 2025AVATAR: Reinforcement Learning to See, Hear, and Reason Over Videocited by 9
  17. 2026OmniVideo-R1: Reinforcing Audio-visual Reasoning with Query Intention and Modality Attentioncited by 9
  18. 2025VGGSounder: Audio-Visual Evaluations for Foundation Modelscited by 8
  19. 2025HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses through Reasoning MLLMscited by 8
  20. 2025OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucinationcited by 8
  21. 2026LiViBench: An Omnimodal Benchmark for Interactive Livestream Video Understandingcited by 8
  22. 2026Demo-ICL: In-Context Learning for Procedural Video Knowledge Acquisitioncited by 8
  23. 2025AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoningcited by 7
  24. 2025MGM-Omni: Scaling Omni LLMs to Personalized Long-Horizon Speechcited by 7
  25. 2026Script-a-Video: Deep Structured Audio-visual Captions via Factorized Streams and Relational Groundingcited by 6
  26. 2025OmniGuard: Unified Omni-Modal Guardrails with Deliberate Reasoningcited by 5
  27. 2025ChronusOmni: Improving Time Awareness of Omni Large Language Modelscited by 5
  28. 2025DeepSound-V1: Start to Think Step-by-Step in the Audio Generation from Videoscited by 4
  29. 2025TokenFLEX: Unified VLM Training for Flexible Visual Tokens Inferencecited by 4
  30. 2025Is Extending Modality The Right Path Towards Omni-Modality?cited by 4
  31. 2025EchoingPixels: Aliasing-Resistant Joint Token Reduction for Audio-Visual LLMscited by 4
  32. 2026ThinkOmni: Lifting Textual Reasoning to Omni-modal Scenarios via Guidance Decodingcited by 4
  33. 2025Multimodal Large Language Models for End-to-End Affective Computing: Benchmarking and Boosting with Generative Knowledge Promptingcited by 3
  34. 2025LongInsightBench: A Comprehensive Benchmark for Evaluating Omni-Modal Models on Human-Centric Long-Video Understandingcited by 3
  35. 2026MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documentscited by 3
  36. 2026Native Active Perception as Reasoning for Omni-Modal Understandingcited by 3
  37. 2025Towards LLM-Centric Multimodal Fusion: A Survey on Integration Strategies and Techniquescited by 2
  38. 2025E3RG: Building Explicit Emotion-driven Empathetic Response Generation System with Multimodal Large Language Modelcited by 2

50 most cited of 79 All 79 (JSON)