mlsanityData to 14 Oct 2026

OmniBench: Towards The Future of Universal Omni-Language Models

arXiv 2409.15272 ↗First posted 23 Sep 2024cs.CL cs.AI cs.CVcited by 102

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Recent advancements in multimodal large language models (MLLMs) have aimed to integrate and interpret data across diverse modalities. However, the capacity of these models to concurrently process and reason about multiple modalities remains underexplored, partly due to the lack of comprehensive modality-wise benchmarks. We introduce OmniBench, a novel benchmark designed to rigorously evaluate models' ability to recognize, interpret, and reason across visual, acoustic, and textual inputs simultaneously. We define language models capable of such tri-modal processing as the omni-language models (OLMs). OmniBench is distinguished by high-quality human annotations, ensuring that accurate responses require integrated understanding and reasoning across all three modalities. Our main findings reveal that: i) open-source OLMs exhibit critical limitations in instruction-following and reasoning capabilities within tri-modal contexts; and ii) most baselines models perform poorly (below 50% accuracy) even when provided with alternative textual representations of images or/and audio. These results suggest that the ability to construct a consistent context from text, image, and audio is often overlooked in existing MLLM training paradigms. To address this gap, we curate an instruction tuning dataset of 84.5K training samples, OmniInstruct, for training OLMs to adapt to tri-modal contexts. We advocate for future research to focus on developing more robust tri-modal integration techniques and training strategies to enhance OLMs. Codes, data and live leaderboard could be found at https://m-a-p.ai/OmniBench.

Key facts. Cited by 102 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 77 references that are in the map. First posted on arXiv 23 Sep 2024.

Builds on

53 of 77 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  2. 2024A Survey on Benchmarks of Multimodal Large Language Models16 Aug
  3. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLM9 Aug
  4. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  5. 2024MANTIS: Interleaved Multi-Image Instruction Tuning2 May
  6. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  7. 2024Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Models18 Apr
  8. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD9 Apr
  9. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling19 Feb
  10. 2024CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark22 Jan
  11. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  12. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  8. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  9. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  11. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  12. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
29 more
  1. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  2. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  3. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  4. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 388
  5. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 373
  6. 2019Clotho: An Audio Captioning Datasetcited by 358
  7. 2024What matters when building vision-language models?cited by 326
  8. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  9. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 220
  10. 2023Listen, Think, and Understandcited by 209
  11. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 191
  12. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 186
  13. 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 181
  14. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
  15. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 140
  16. 2018AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scalecited by 128
  17. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 102
  18. 2021Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videoscited by 86
  19. 2023BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writingcited by 79
  20. 2023Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioningcited by 77
  21. 2023Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosiscited by 77
  22. 2022Vocalsound: A Dataset for Improving Human Vocal Sounds Recognitioncited by 62
  23. 2023MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Responsecited by 45
  24. 2024Foundation Models for Music: A Surveycited by 37
  25. 2023Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answeringcited by 29
  26. 2024SciMMIR: Benchmarking Scientific Multi-modal Information Retrievalcited by 23
  27. 2022Learning from Taxonomy: Multi-label Few-Shot Classification for Everyday Sound Recognitioncited by 8
  28. 2024Mind the Domain Gap: a Systematic Analysis on Bioacoustic Sound Event Detectioncited by 7
  29. 2021Emergency Vehicles Audio Detection and Localization in Autonomous Drivingcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 50; 2026: 49.

102 papers.

Sort:

By area:

  1. 2025Qwen2.5-Omni Technical Reportcited by 736
  2. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 182
  3. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 155
  4. 2025Baichuan-Omni-1.5 Technical Reportcited by 111
  5. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 110
  6. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  7. 2025Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Modelcited by 62
  8. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMscited by 59
  9. 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 59
  10. 2025HumanOmniV2: From Understanding to Omni-Modal Reasoning with Contextcited by 55
  11. 2025OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contextscited by 51
  12. 2024AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?cited by 47
38 more
  1. 2025Judge Anything: MLLM as a Judge Across Any Modalitycited by 34
  2. 2025AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decodingcited by 30
  3. 2025EchoInk-R1: Exploring Audio-Visual Reasoning in Multimodal LLMs via Reinforcement Learningcited by 29
  4. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 29
  5. 2025ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Modelscited by 27
  6. 2025LongCat-Flash-Omni Technical Reportcited by 26
  7. 2024MIO: A Foundation Model on Multimodal Tokenscited by 24
  8. 2026LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMscited by 24
  9. 2026OmniGAIA: Towards Native Omni-Modal AI Agentscited by 22
  10. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 21
  11. 2025Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Datacited by 21
  12. 2025See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Modelscited by 20
  13. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgettingcited by 18
  14. 2025Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Modelscited by 18
  15. 2025MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLMcited by 17
  16. 2025CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Followingcited by 17
  17. 2025JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluationcited by 17
  18. 2025AV-Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMscited by 16
  19. 2025Empowering Time Series Analysis with Synthetic Data: A Survey and Outlook in the Era of Foundation Modelscited by 15
  20. 2026MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videoscited by 14
  21. 2025OmniEval: A Benchmark for Evaluating Omni-modal Models with Visual, Auditory, and Textual Inputscited by 13
  22. 2025UNO-Bench: A Unified Benchmark for Exploring the Compositional Law Between Uni-modal and Omni-modal in Omni Modelscited by 13
  23. 2026SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Modelscited by 13
  24. 2025MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystackscited by 12
  25. 2025AVATAR: Reinforcement Learning to See, Hear, and Reason Over Videocited by 12
  26. 2024VideoCogQA: A Controllable Benchmark for Evaluating Cognitive Abilities in Video-Language Modelscited by 10
  27. 2025OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-Time Self-Aware Emotional Speech Synthesiscited by 10
  28. 2025ColorBench: Benchmarking Mobile Agents with Graph-Structured Framework for Complex Long-Horizon Taskscited by 10
  29. 2025HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses through Reasoning MLLMscited by 9
  30. 2025AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoningcited by 8
  31. 2025AutoMV: An Automatic Multi-Agent System for Music Video Generationcited by 8
  32. 2026Towards Multimodal Lifelong Understanding: A Dataset and Agentic Baselinecited by 8
  33. 2025MAVERIX: Multimodal Audio-Visual Evaluation and Recognition IndeXcited by 7
  34. 2025MultiVox: A Benchmark for Evaluating Voice Assistants for Multimodal Interactionscited by 7
  35. 2026ThinkOmni: Lifting Textual Reasoning to Omni-modal Scenarios via Guidance Decodingcited by 7
  36. 2025XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Modelscited by 6
  37. 2026The Interspeech 2026 Audio Reasoning Challenge: Evaluating Reasoning Process Quality for Audio Reasoning Models and Agentscited by 6
  38. 2026On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMscited by 6

50 most cited of 102 All 102 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024OmnixR: Evaluating Omni-modality Language Models on Reasoning across Modalities
  2. 2026Boosting Omni-Modal Language Models: Staged Post-Training with Visually Debiased Evaluation
  3. 2025OMHBench: Benchmarking Balanced and Grounded Omni-Modal Multi-Hop Reasoning
  4. 2026LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning
  5. 2026C3PO: Evaluating Cross-Modal Composition and Counterfactual Performance in Omnimodal Models
  6. 2026OmniCap-IF: Benchmarking and Improving Instruction Following Abilities for Omni-Video Captioning
  7. 2026Omni-Judge: Can Omni-LLMs Serve as Human-Aligned Judges for Text-Conditioned Audio-Video Generation?
  8. 2026Can MiniMax-H3 Reason About the Physical World? An Evaluation of Omni-Modal Generative Model