# Audio-centric Video Understanding Benchmark without Text Shortcut

arXiv: https://arxiv.org/abs/2503.19951 · HTML: https://mlsanity.com/paper/2503.19951/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 34 of its 49 references that are in the map.
- First posted on arXiv 25 Mar 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Builds on (most cited first, 30 of 34)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [VILA: On Pre-training for Visual Language Models](https://mlsanity.com/paper/2312.07533/) (2023)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [NExT-GPT: Any-to-Any Multimodal LLM](https://mlsanity.com/paper/2309.05519/) (2023)
- [TempCompass: Do Video LLMs Really Understand Videos?](https://mlsanity.com/paper/2403.00476/) (2024)
- [PandaGPT: One Model To Instruction-Follow Them All](https://mlsanity.com/paper/2305.16355/) (2023)
- [LongVILA: Scaling Long-Context Visual Language Models for Long Videos](https://mlsanity.com/paper/2408.10188/) (2024)
- [Learning to Answer Questions in Dynamic Audio-Visual Scenarios](https://mlsanity.com/paper/2203.14072/) (2022)
- [OneLLM: One Framework to Align All Modalities with Language](https://mlsanity.com/paper/2312.03700/) (2023)
- [MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding](https://mlsanity.com/paper/2406.14515/) (2024)
- [VITA: Towards Open-Source Interactive Omni Multimodal LLM](https://mlsanity.com/paper/2408.05211/) (2024)
- [VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset](https://mlsanity.com/paper/2305.18500/) (2023)
- [VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset](https://mlsanity.com/paper/2304.08345/) (2023)
- [Audio-Visual Scene-Aware Dialog](https://mlsanity.com/paper/1901.09107/) (2019)
- [TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models](https://mlsanity.com/paper/2410.10818/) (2024)
- [VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation](https://mlsanity.com/paper/2106.04632/) (2021)
- [Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videos](https://mlsanity.com/paper/2110.05122/) (2021)
- [VideoVista: A Versatile Benchmark for Video Understanding and Reasoning](https://mlsanity.com/paper/2406.11303/) (2024)
- [VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Models](https://mlsanity.com/paper/2311.17404/) (2023)

## Built on by (most cited first, 30 of 36)

- [Qwen3.5-Omni Technical Report](https://mlsanity.com/paper/2604.15804/) (2026)
- [Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models](https://mlsanity.com/paper/2505.04921/) (2025)
- [MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction](https://mlsanity.com/paper/2604.27393/) (2026)
- [video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Models](https://mlsanity.com/paper/2506.15220/) (2025)
- [VISTA: A Test-Time Self-Improving Video Generation Agent](https://mlsanity.com/paper/2510.15831/) (2025)
- [LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMs](https://mlsanity.com/paper/2603.19217/) (2026)
- [OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Models](https://mlsanity.com/paper/2511.14582/) (2025)
- [JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation](https://mlsanity.com/paper/2512.12772/) (2025)
- [Active Perception Agent for Omnimodal Audio-Video Understanding](https://mlsanity.com/paper/2512.23646/) (2025)
- [FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMs](https://mlsanity.com/paper/2601.13836/) (2026)
- [D-ORCA: Dialogue-Centric Optimization for Robust Audio-Visual Captioning](https://mlsanity.com/paper/2602.07960/) (2026)
- [On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMs](https://mlsanity.com/paper/2603.14337/) (2026)
- [DASH: Dynamic Audio-Driven Semantic Chunking for Efficient Omnimodal Token Compression](https://mlsanity.com/paper/2603.15685/) (2026)
- [MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models](https://mlsanity.com/paper/2601.21181/) (2026)
- [Beyond Closed-Pool Video Retrieval: A Benchmark and Agent Framework for Real-World Video Search and Moment Localization](https://mlsanity.com/paper/2602.10159/) (2026)
- [OmniRefine: Alignment-Aware Cooperative Compression for Efficient Omnimodal Large Language Models](https://mlsanity.com/paper/2605.12056/) (2026)
- [Sandboxed Coding Agents are Competitive Omni-modal Task Solvers](https://mlsanity.com/paper/2606.00579/) (2026)
- [OmniPack: Unified Token Compression for Efficient Omni-modal Large Language Models](https://mlsanity.com/paper/2608.03812/) (2026)
- [Do Modern Video-LLMs Need to Listen? A Benchmark Audit and Scalable Remedy](https://mlsanity.com/paper/2509.17901/) (2025)
- [Protecting Bystander Privacy via Selective Hearing in Audio LLMs](https://mlsanity.com/paper/2512.06380/) (2025)
- [C3PO: Evaluating Cross-Modal Composition and Counterfactual Performance in Omnimodal Models](https://mlsanity.com/paper/2608.05381/) (2026)
- [Deferred Audio Pruning with Local Audio-Visual Dynamics for Omni-LLMs](https://mlsanity.com/paper/2608.08794/) (2026)
- [TLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streaming](https://mlsanity.com/paper/2608.20958/) (2026)
- [Qwen3.8-Omni: Towards Native Omni-Modal Agents](https://mlsanity.com/paper/2609.25611/) (2026)
- [Cross-Modal Coreference Alignment: Enabling Reliable Information Transfer in Omni-LLMs](https://mlsanity.com/paper/2604.05522/) (2026)
- [OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments](https://mlsanity.com/paper/2605.18758/) (2026)
- [From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMs](https://mlsanity.com/paper/2606.10147/) (2026)
- [OmniDelta: Skill-Driven Budget Allocation for Token Compression in OmniLLMs](https://mlsanity.com/paper/2607.25669/) (2026)
- [Allocation Before Ranking: Decoupled Token Compression for OmniLLMs](https://mlsanity.com/paper/2608.01665/) (2026)
- [V-LynX: Token Interface Alignment for Video+X LLMs](https://mlsanity.com/paper/2606.00508/) (2026)

## Abstract (arXiv)

Audio often serves as an auxiliary modality in video understanding tasks of audio-visual large language models (LLMs), merely assisting in the comprehension of visual information. However, a thorough understanding of videos significantly depends on auditory information, as audio offers critical context, emotional cues, and semantic meaning that visual data alone often lacks. This paper proposes an audio-centric video understanding benchmark (AVUT) to evaluate the video comprehension capabilities of multimodal LLMs with a particular focus on auditory information. AVUT introduces a suite of carefully designed audio-centric tasks, holistically testing the understanding of both audio content and audio-visual interactions in videos. Moreover, this work points out the text shortcut problem that largely exists in other benchmarks where the correct answer can be found from question text alone without needing videos. AVUT addresses this problem by proposing a answer permutation-based filtering mechanism. A thorough evaluation across a diverse range of open-source and proprietary multimodal LLMs is performed, followed by the analyses of deficiencies in audio-visual LLMs. Demos and data are available at https://github.com/lark-png/AVUT.