mlsanityData to 7 Oct 2026

AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?

arXiv 2412.02611 ↗First posted 3 Dec 2024cs.CV cs.AI cs.CLcited by 46

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Recently, multimodal large language models (MLLMs), such as GPT-4o, Gemini 1.5 Pro, and Reka Core, have expanded their capabilities to include vision and audio modalities. While these models demonstrate impressive performance across a wide range of audio-visual applications, our proposed DeafTest reveals that MLLMs often struggle with simple tasks humans find trivial: 1) determining which of two sounds is louder, and 2) determining which of two sounds has a higher pitch. Motivated by these observations, we introduce AV-Odyssey Bench, a comprehensive audio-visual benchmark designed to assess whether those MLLMs can truly understand the audio-visual information. This benchmark encompasses 4,555 carefully crafted problems, each incorporating text, visual, and audio components. To successfully infer answers, models must effectively leverage clues from both visual and audio inputs. To ensure precise and objective evaluation of MLLM responses, we have structured the questions as multiple-choice, eliminating the need for human evaluation or LLM-assisted assessment. We benchmark a series of closed-source and open-source models and summarize the observations. By revealing the limitations of current models, we aim to provide useful insight for future dataset collection and model development.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 68 of its 88 references that are in the map. First posted on arXiv 3 Dec 2024.

Builds on

68 of 88 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding22 Oct
  2. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Input28 Aug
  3. 2024Long Context Transfer from Language to Vision24 Jun
  4. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding4 Dec
  5. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models28 Nov
  6. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 62

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2024GPT-4o System Cardcited by 4,467
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
38 more
  1. 2017The Kinetics Human Action Video Datasetcited by 1,894
  2. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  3. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  4. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  5. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  6. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  7. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  8. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  9. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  10. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  11. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  12. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  13. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  14. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  15. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  16. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  17. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  18. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  19. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  20. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  21. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  22. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  23. 2024Qwen2-Audio Technical Reportcited by 543
  24. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  25. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  26. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 376
  27. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  28. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  29. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  30. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  31. 2021GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoningcited by 224
  32. 2016FMA: A Dataset For Music Analysiscited by 205
  33. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 188
  34. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  35. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 172
  36. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 144
  37. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 144
  38. 2022UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expressioncited by 137

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 24; 2026: 22.

46 papers.

Sort:

By area:

  1. 2025Qwen2.5-Omni Technical Reportcited by 704
  2. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  3. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 147
  4. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 101
  5. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMscited by 54
  6. 2025MME-Reasoning: A Comprehensive Benchmark for Logical Reasoning in MLLMscited by 31
  7. 2026LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMscited by 22
  8. 2026OmniGAIA: Towards Native Omni-Modal AI Agentscited by 21
  9. 2025AVTrustBench: Assessing and Enhancing Reliability and Robustness in Audio-Visual LLMscited by 19
  10. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 19
  11. 2025See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Modelscited by 19
  12. 2025AV-Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMscited by 15
34 more
  1. 2026MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videoscited by 14
  2. 2025JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluationcited by 13
  3. 2025UNO-Bench: A Unified Benchmark for Exploring the Compositional Law Between Uni-modal and Omni-modal in Omni Modelscited by 12
  4. 2025MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystackscited by 11
  5. 2025OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-Time Self-Aware Emotional Speech Synthesiscited by 10
  6. 2025Aurelia: Test-time Reasoning Distillation in Audio-Visual LLMscited by 9
  7. 2025AVATAR: Reinforcement Learning to See, Hear, and Reason Over Videocited by 9
  8. 2025MAVERIX: Multimodal Audio-Visual Evaluation and Recognition IndeXcited by 7
  9. 2025AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoningcited by 7
  10. 2026Omni-Modal Dissonance Benchmark: Systematically Breaking Modality Consensus to Probe Robustness and Calibrated Abstentioncited by 6
  11. 2025FysicsWorld: A Unified Full-Modality Benchmark for Any-to-Any Understanding, Generation, and Reasoningcited by 5
  12. 2026AVMeme Exam: A Multimodal Multilingual Multicultural Benchmark for LLMs' Contextual and Cultural Knowledge and Thinkingcited by 5
  13. 2025Learning Sparsity for Effective and Efficient Music Performance Question Answeringcited by 4
  14. 2025XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Modelscited by 4
  15. 2025TELEVAL: A Benchmark Designed for Spoken Language Models in Chinese Interactive Scenarioscited by 2
  16. 2026Sandboxed Coding Agents are Competitive Omni-modal Task Solverscited by 2
  17. 2026Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videoscited by 2
  18. 2025AuditoryBench++: Can Language Models Understand Auditory Knowledge without Hearing?cited by 1
  19. 2025Do Modern Video-LLMs Need to Listen? A Benchmark Audit and Scalable Remedycited by 1
  20. 2026TraceAV-Bench: Benchmarking Multi-Hop Trajectory Reasoning over Long Audio-Visual Videoscited by 1
  21. 2026Omni-DeepSearch: A Benchmark for Audio-Driven Omni-Modal Deep Searchcited by 1
  22. 2026VideoOdyssey: A Benchmark for Ultra-Long-Context and Omni-Modal Video Understandingcited by 1
  23. 2026SVHalluc: Benchmarking Speech-Vision Hallucination in Audio-Visual Large Language Modelscited by 1
  24. 2026From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMscited by 1
  25. 2026MuseBench: Benchmarking Intent-Level Audiovisual Arts Understanding in MLLMscited by 1
  26. 2026Chain of Modality: From Static Fusion to Dynamic Orchestration in Omni-MLLMscited by 0
  27. 2026Boosting Omni-Modal Language Models: Staged Post-Training with Visually Debiased Evaluationcited by 0
  28. 2026AVI-Bench: Toward Human-like Audio-Visual Intelligence of Omni-MLLMscited by 0
  29. 2026CogniRoute: Learning to Route Social Evidence in Omni-Modal Modelscited by 0
  30. 2026SafeNexus: Discovering and Steering Modality-Universal Safety Neurons in MLLMscited by 0
  31. 2026Training-Free Speech-Centric Omni Understanding with Frozen VLMscited by 0
  32. 2026OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialoguecited by 0
  33. 2026OmniFysics-Nano-V2 Technical Report: Understanding the Physical World Across Modalitiescited by 0
  34. 2026SAVU-BENCH: A Real-World Benchmark for Spatial Audio-Visual Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Do Audio-Visual Large Language Models Really See and Hear?
  2. 2024What do MLLMs hear? Examining reasoning with text and sound components in Multimodal Large Language Models
  3. 2025Multifaceted Evaluation of Audio-Visual Capability for MLLMs: Effectiveness, Efficiency, Generalizability and Robustness
  4. 2024Large Language Models are Strong Audio-Visual Speech Recognition Learners
  5. 2025When Eyes and Ears Disagree: Can MLLMs Discern Audio-Visual Confusion?
  6. 2026DEAF: A Benchmark for Diagnostic Evaluation of Acoustic Faithfulness in Audio Language Models
  7. 2025Bridging Ears and Eyes: Analyzing Audio and Visual Large Language Models to Humans in Visible Sound Recognition and Reducing Their Sensory Gap via Cross-Modal Distillation
  8. 2024A Survey on Benchmarks of Multimodal Large Language Models