mlsanityData to 7 Oct 2026

Do Audio-Visual Large Language Models Really See and Hear?

arXiv 2604.02605 ↗First posted 3 Apr 2026cs.AI cs.SDcited by 9

Not on the map. Why?

Abstract (arXiv)

Audio-Visual Large Language Models (AVLLMs) are emerging as unified interfaces to multimodal perception. We present the first mechanistic interpretability study of AVLLMs, analyzing how audio and visual features evolve and fuse through different layers of an AVLLM to produce the final text outputs. We find that although AVLLMs encode rich audio semantics at intermediate layers, these capabilities largely fail to surface in the final text generation when audio conflicts with vision. Probing analyses show that useful latent audio information is present, but deeper fusion layers disproportionately privilege visual representations that tend to suppress audio cues. We further trace this imbalance to training: the AVLLM's audio behavior strongly matches its vision-language base model, indicating limited additional alignment to audio supervision. Our findings reveal a fundamental modality bias in AVLLMs and provide new mechanistic insights into how multimodal LLMs integrate audio and vision.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 48 of its 63 references that are in the map. First posted on arXiv 3 Apr 2026.

Builds on

48 of 63 references.

References · 48

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  5. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  6. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  7. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  8. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  9. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  10. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  11. 2025Qwen2.5-Omni Technical Reportcited by 704
  12. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
36 more
  1. 2018Object Hallucination in Image Captioningcited by 661
  2. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  3. 2019Clotho: An Audio Captioning Datasetcited by 354
  4. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  5. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  6. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  7. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 275
  8. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  9. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 186
  10. 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 183
  11. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  12. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 147
  13. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  14. 2024Towards Interpreting Visual Information Processing in Vision-Language Modelscited by 115
  15. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 101
  16. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 101
  17. 2024Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?cited by 68
  18. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Modelscited by 66
  19. 2024Cross-modal Information Flow in Multimodal Large Language Modelscited by 65
  20. 2023RECALL: A Benchmark for LLMs Robustness against External Counterfactual Knowledgecited by 51
  21. 2024AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Modelscited by 50
  22. 2024VLind-Bench: Measuring Language Priors in Large Vision-Language Modelscited by 44
  23. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 44
  24. 2023CLAIR: Evaluating Image Captions with Large Language Modelscited by 43
  25. 2022Probing Classifiers are Unreliable for Concept Removal and Detectioncited by 40
  26. 2023Text encoders bottleneck compositionality in contrastive vision-language modelscited by 34
  27. 2025OmniVLA: An Omni-Modal Vision-Language-Action Model for Robot Navigationcited by 34
  28. 2024SFR-RAG: Towards Contextually Faithful LLMscited by 19
  29. 2025AVTrustBench: Assessing and Enhancing Reliability and Robustness in Audio-Visual LLMscited by 19
  30. 2025AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Modelscited by 18
  31. 2024MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluationcited by 17
  32. 2024OmnixR: Evaluating Omni-modality Language Models on Reasoning across Modalitiescited by 13
  33. 2024Is Your Large Language Model Knowledgeable or a Choices-Only Cheater?cited by 12
  34. 2024Why do LLaVA Vision-Language Models Reply to Images in English?cited by 11
  35. 2025RoboEgo System Card: An Omnimodal Model with Native Full Duplexitycited by 5
  36. 2025Learning Interpretable Features in Audio Latent Spaces via Sparse Autoencoderscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 9.

9 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026Probing Cross-modal Information Hubs in Audio-Visual LLMs
  2. 2025Fork-Merge Decoding: Enhancing Multimodal Understanding in Audio-Visual Large Language Models
  3. 2026AVI-Bench: Toward Human-like Audio-Visual Intelligence of Omni-MLLMs
  4. 2025Bridging Ears and Eyes: Analyzing Audio and Visual Large Language Models to Humans in Visible Sound Recognition and Reducing Their Sensory Gap via Cross-Modal Distillation
  5. 2024AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?
  6. 2023Audio-Visual LLM for Video Understanding
  7. 2025Nexus: An Omni-Perceptive And -Interactive Model for Language, Audio, And Vision
  8. 2024Meerkat: Audio-Visual Large Language Model for Grounding in Space and Time