mlsanityData to 14 Oct 2026

Multifaceted Evaluation of Audio-Visual Capability for MLLMs: Effectiveness, Efficiency, Generalizability and Robustness

arXiv 2504.16936 ↗First posted 3 Apr 2025cs.MM cs.CV cs.SDcited by 3

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Multi-modal large language models (MLLMs) have recently achieved great success in processing and understanding information from diverse modalities (e.g., text, audio, and visual signals). Despite their growing popularity, there remains a lack of comprehensive evaluation measuring the audio-visual capabilities of these models, especially in diverse scenarios (e.g., distribution shifts and adversarial attacks). In this paper, we present a multifaceted evaluation of the audio-visual capability of MLLMs, focusing on four key dimensions: effectiveness, efficiency, generalizability, and robustness. Through extensive experiments, we find that MLLMs exhibit strong zero-shot and few-shot generalization abilities, enabling them to achieve great performance with limited data. However, their success relies heavily on the vision modality, which impairs performance when visual input is corrupted or missing. Additionally, while MLLMs are susceptible to adversarial samples, they demonstrate greater robustness compared to traditional models. The experimental results and our findings provide insights into the audio-visual capabilities of MLLMs, highlighting areas for improvement and offering guidance for future research.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 56 of its 91 references that are in the map. First posted on arXiv 3 Apr 2025.

Builds on

56 of 91 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios7 Mar
  2. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov
  3. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 53

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2013Intriguing properties of neural networkscited by 5,155
  5. 2017The Kinetics Human Action Video Datasetcited by 1,909
  6. 2021ViViT: A Video Vision Transformercited by 1,620
  7. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  8. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,318
  9. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  10. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  11. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  12. 2021AST: Audio Spectrogram Transformercited by 643
38 more
  1. 2022Efficient Test-Time Model Adaptation without Forgettingcited by 469
  2. 2023Towards Stable Test-Time Adaptation in Dynamic Wild Worldcited by 415
  3. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  4. 2023A Comprehensive Survey on Test-Time Adaptation under Distribution Shiftscited by 359
  5. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  6. 2018Adversarial Attacks and Defences: A Surveycited by 259
  7. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 217
  8. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 197
  9. 2022Are Multimodal Transformers Robust to Missing Modality?cited by 186
  10. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 173
  11. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 172
  12. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 169
  13. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  14. 2022PMR: Prototypical Modal Rebalance for Multimodal Learningcited by 116
  15. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  16. 2024Entropy is not Enough for Test-Time Adaptation: From the Perspective of Disentangled Factorscited by 106
  17. 2021TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Groundingcited by 86
  18. 2022MM-TTA: Multi-Modal Test-Time Adaptation for 3D Semantic Segmentationcited by 76
  19. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 76
  20. 2022Audio Self-supervised Learning: A Surveycited by 65
  21. 2023PerceptionGPT: Effectively Fusing Visual Perception into LLMcited by 59
  22. 2022Target-Driven Structured Transformer Planner for Vision-Language Navigationcited by 58
  23. 2023TouchStone: Evaluating Vision-Language Models by Language Modelscited by 55
  24. 2023Dense-Localizing Audio-Visual Events in Untrimmed Videos: A Large-Scale Benchmark and Baselinecited by 46
  25. 2023Towards Open-Set Test-Time Adaptation Utilizing the Wisdom of Crowds in Entropy Minimizationcited by 39
  26. 2022TeST: Test-time Self-Training under Distribution Shiftcited by 29
  27. 2024DTLLM-VLT: Diverse Text Generation for Visual Language Tracking Based on LLMcited by 24
  28. 2020Object Hider: Adversarial Patch Attack Against Object Detectorscited by 23
  29. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 23
  30. 2024Personalizing Multimodal Large Language Models for Image Captioning: An Experimental Analysiscited by 23
  31. 2024LOVA3: Learning to Visual Question Answering, Asking and Assessmentcited by 21
  32. 2024Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmarkcited by 20
  33. 2023AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Modelscited by 19
  34. 2024MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluationcited by 18
  35. 2025Towards Robust Multimodal Open-set Test-time Adaptation via Adaptive Entropy-aware Optimizationcited by 12
  36. 2024MLLM-SUL: Multimodal Large Language Model for Semantic Scene Understanding and Localization in Traffic Scenarioscited by 8
  37. 2023Overcome Modal Bias in Multi-modal Federated Learning via Balanced Modality Selectioncited by 6
  38. 2021Emergency Vehicles Audio Detection and Localization in Autonomous Drivingcited by 5

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Dynamic Bundling with Large Language Models for Zero-Shot Inference on Text-Attributed Graphscited by 3
  2. 2026Omni-DeepSearch: A Benchmark for Audio-Driven Omni-Modal Deep Searchcited by 1
  3. 2026MMClima: A Framework for Multimodal Climate Science Data and Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Investigating Vulnerabilities and Defenses Against Audio-Visual Attacks: A Comprehensive Survey Emphasizing Multimodal Models
  2. 2025Survey of Adversarial Robustness in Multimodal Large Language Models
  3. 2024AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?
  4. 2026Do Audio-Visual Large Language Models Really See and Hear?
  5. 2024The Instinctive Bias: Spurious Images lead to Illusion in MLLMs
  6. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
  7. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  8. 2026Investigating Adversarial Robustness of Multi-modal Large Language Models