mlsanityData to 7 Oct 2026

Audio-centric Video Understanding Benchmark without Text Shortcut

arXiv 2503.19951 ↗First posted 25 Mar 2025cs.CV cs.AIcited by 33

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Audio often serves as an auxiliary modality in video understanding tasks of audio-visual large language models (LLMs), merely assisting in the comprehension of visual information. However, a thorough understanding of videos significantly depends on auditory information, as audio offers critical context, emotional cues, and semantic meaning that visual data alone often lacks. This paper proposes an audio-centric video understanding benchmark (AVUT) to evaluate the video comprehension capabilities of multimodal LLMs with a particular focus on auditory information. AVUT introduces a suite of carefully designed audio-centric tasks, holistically testing the understanding of both audio content and audio-visual interactions in videos. Moreover, this work points out the text shortcut problem that largely exists in other benchmarks where the correct answer can be found from question text alone without needing videos. AVUT addresses this problem by proposing a answer permutation-based filtering mechanism. A thorough evaluation across a diverse range of open-source and proprietary multimodal LLMs is performed, followed by the analyses of deficiencies in audio-visual LLMs. Demos and data are available at https://github.com/lark-png/AVUT.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 49 references that are in the map. First posted on arXiv 25 Mar 2025.

Builds on

34 of 49 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videos19 Aug
  2. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding20 Jun
  3. 2024VideoVista: A Versatile Benchmark for Video Understanding and Reasoning17 Jun
  4. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis31 May
  5. 2024CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios7 Mar
  6. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark28 Nov
  7. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  6. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  7. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  8. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  9. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  10. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  11. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  12. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
14 more
  1. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  2. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 188
  3. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  4. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 172
  5. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  6. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  7. 2019Audio-Visual Scene-Aware Dialogcited by 122
  8. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Modelscited by 96
  9. 2021VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluationcited by 95
  10. 2021Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videoscited by 86
  11. 2023VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Modelscited by 55
  12. 2024LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videoscited by 49
  13. 2023Dense-Localizing Audio-Visual Events in Untrimmed Videos: A Large-Scale Benchmark and Baselinecited by 45
  14. 2017See, Hear, and Read: Deep Aligned Representationscited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 25.

33 papers.

Sort:

By area:

  1. 2026Qwen3.5-Omni Technical Reportcited by 146
  2. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  3. 2026MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interactioncited by 64
  4. 2025video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Modelscited by 39
  5. 2025VISTA: A Test-Time Self-Improving Video Generation Agentcited by 26
  6. 2026LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMscited by 22
  7. 2025OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Modelscited by 21
  8. 2025JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluationcited by 13
  9. 2025Active Perception Agent for Omnimodal Audio-Video Understandingcited by 13
  10. 2026FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMscited by 7
  11. 2026D-ORCA: Dialogue-Centric Optimization for Robust Audio-Visual Captioningcited by 6
  12. 2026DASH: Dynamic Audio-Driven Semantic Chunking for Efficient Omnimodal Token Compressioncited by 6
21 more
  1. 2026On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMscited by 5
  2. 2026OmniPack: Unified Token Compression for Efficient Omni-modal Large Language Modelscited by 3
  3. 2026MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Modelscited by 2
  4. 2026OmniRefine: Alignment-Aware Cooperative Compression for Efficient Omnimodal Large Language Modelscited by 2
  5. 2026Sandboxed Coding Agents are Competitive Omni-modal Task Solverscited by 2
  6. 2026Deferred Audio Pruning with Local Audio-Visual Dynamics for Omni-LLMscited by 2
  7. 2026TLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streamingcited by 2
  8. 2026Qwen3.8-Omni: Towards Native Omni-Modal Agentscited by 2
  9. 2025Do Modern Video-LLMs Need to Listen? A Benchmark Audit and Scalable Remedycited by 1
  10. 2025Protecting Bystander Privacy via Selective Hearing in Audio LLMscited by 1
  11. 2026Beyond Closed-Pool Video Retrieval: A Benchmark and Agent Framework for Real-World Video Search and Moment Localizationcited by 1
  12. 2026OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environmentscited by 1
  13. 2026From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMscited by 1
  14. 2026OmniDelta: Skill-Driven Budget Allocation for Token Compression in OmniLLMscited by 1
  15. 2026Allocation Before Ranking: Decoupled Token Compression for OmniLLMscited by 1
  16. 2026C3PO: Evaluating Cross-Modal Composition and Counterfactual Performance in Omnimodal Modelscited by 1
  17. 2026Cross-Modal Coreference Alignment: Enabling Reliable Information Transfer in Omni-LLMscited by 0
  18. 2026V-LynX: Token Interface Alignment for Video+X LLMscited by 0
  19. 2026video-SALMONN-R3: Learning to ReWatch, ReAsk, and ReAnswer for Efficient Video Understandingcited by 0
  20. 2026Training-Free Speech-Centric Omni Understanding with Frozen VLMscited by 0
  21. 2026SyncRA: Learning Temporal Correspondence in Omni-Modal Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Audio-Visual LLM for Video Understanding
  2. 2024Audio-visual training for improved grounding in video-text LLMs
  3. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs
  4. 2025See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Models
  5. 2026EXAM2: Extending Audio Understanding in Multilingual and Multimodal Analysis
  6. 2025Aligned Better, Listen Better for Audio-Visual Large Language Models
  7. 2026MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos
  8. 2025AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoning