# Do Modern Video-LLMs Need to Listen? A Benchmark Audit and Scalable Remedy

arXiv: https://arxiv.org/abs/2509.17901 · HTML: https://mlsanity.com/paper/2509.17901/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 35 references that are in the map.
- First posted on arXiv 22 Sep 2025.

## Builds on (most cited first, 30 of 31)

- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025)
- [LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding](https://mlsanity.com/paper/2407.15754/) (2024)
- [LLaVA-Video: Video Instruction Tuning With Synthetic Data](https://mlsanity.com/paper/2410.02713/) (2024)
- [ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering](https://mlsanity.com/paper/1906.02467/) (2019)
- [Qwen2-Audio Technical Report](https://mlsanity.com/paper/2407.10759/) (2024)
- [TempCompass: Do Video LLMs Really Understand Videos?](https://mlsanity.com/paper/2403.00476/) (2024)
- [LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding](https://mlsanity.com/paper/2410.17434/) (2024)
- [VideoMamba: State Space Model for Efficient Video Understanding](https://mlsanity.com/paper/2403.06977/) (2024)
- [Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos](https://mlsanity.com/paper/2501.13826/) (2025)
- [Learning to Answer Questions in Dynamic Audio-Visual Scenarios](https://mlsanity.com/paper/2203.14072/) (2022)
- [WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMs](https://mlsanity.com/paper/2502.04326/) (2025)
- [Audio-Visual Scene-Aware Dialog](https://mlsanity.com/paper/1901.09107/) (2019)
- [OmniBench: Towards The Future of Universal Omni-Language Models](https://mlsanity.com/paper/2409.15272/) (2024)
- [AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models](https://mlsanity.com/paper/2410.18325/) (2024)
- [AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?](https://mlsanity.com/paper/2412.02611/) (2024)
- [Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformers](https://mlsanity.com/paper/2503.11579/) (2025)
- [Audio-centric Video Understanding Benchmark without Text Shortcut](https://mlsanity.com/paper/2503.19951/) (2025)
- [Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answering](https://mlsanity.com/paper/2310.06238/) (2023)
- [BIMBA: Selective-Scan Compression for Long-Range Video Question Answering](https://mlsanity.com/paper/2503.09590/) (2025)
- [SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing](https://mlsanity.com/paper/2506.05414/) (2025)
- [On Efficient Language and Vision Assistants for Visually-Situated Natural Language Understanding: What Matters in Reading and Reasoning](https://mlsanity.com/paper/2406.11823/) (2024)

## Built on by (most cited first, 2 of 2)

- [Sandboxed Coding Agents are Competitive Omni-modal Task Solvers](https://mlsanity.com/paper/2606.00579/) (2026)
- [Relevant Evidence Decoding for Audio-Visual Hallucination Mitigation](https://mlsanity.com/paper/2610.02976/) (2026)

## Abstract (arXiv)

Speech and audio encoders developed over years of community effort are routinely excluded from video understanding pipelines, not because they fail, but because benchmarks never required listening. We audit 10 video benchmarks and find items largely solvable from visual cues alone: a single-frame probe answers about 76% of AVQA without audio, suggesting poor measurement of audio-visual reasoning. Building on LLaVA-OneVision, we attach a speech/audio encoder and compare five compressor architectures under 25-fold token reduction (25 Hz to 1 Hz). Across 10 benchmarks, with and without filtering, audio yields clear gains on tasks requiring speech comprehension or cross-modal grounding, while vision-centric suites remain largely unaffected. Our results show that speech encoders play a larger role in video understanding than current benchmarks suggest. We will open-source our work at https://github.com/naver-ai/unimambamia-av.