Do Audio-Visual Large Language Models Really See and Hear?
Not on the map. Why?
Abstract (arXiv)
Audio-Visual Large Language Models (AVLLMs) are emerging as unified interfaces to multimodal perception. We present the first mechanistic interpretability study of AVLLMs, analyzing how audio and visual features evolve and fuse through different layers of an AVLLM to produce the final text outputs. We find that although AVLLMs encode rich audio semantics at intermediate layers, these capabilities largely fail to surface in the final text generation when audio conflicts with vision. Probing analyses show that useful latent audio information is present, but deeper fusion layers disproportionately privilege visual representations that tend to suppress audio cues. We further trace this imbalance to training: the AVLLM's audio behavior strongly matches its vision-language base model, indicating limited additional alignment to audio supervision. Our findings reveal a fundamental modality bias in AVLLMs and provide new mechanistic insights into how multimodal LLMs integrate audio and vision.
Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 48 of its 63 references that are in the map. First posted on arXiv 3 Apr 2026.
Builds on
48 of 63 references.
References · 48
- 2025Qwen3 Technical Reportcited by 8,195
- 2023Visual Instruction Tuningcited by 7,792
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
- 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
- 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
- 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
- 2016Understanding intermediate layers using linear classifier probescited by 1,279
- 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
- 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
- 2025Qwen2.5-Omni Technical Reportcited by 704
- 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
36 more
- 2018Object Hallucination in Image Captioningcited by 661
- 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
- 2019Clotho: An Audio Captioning Datasetcited by 354
- 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
- 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
- 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
- 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 275
- 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
- 2024Transcoders Find Interpretable LLM Feature Circuitscited by 186
- 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 183
- 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
- 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 147
- 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
- 2024Towards Interpreting Visual Information Processing in Vision-Language Modelscited by 115
- 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 101
- 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 101
- 2024Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?cited by 68
- 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Modelscited by 66
- 2024Cross-modal Information Flow in Multimodal Large Language Modelscited by 65
- 2023RECALL: A Benchmark for LLMs Robustness against External Counterfactual Knowledgecited by 51
- 2024AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Modelscited by 50
- 2024VLind-Bench: Measuring Language Priors in Large Vision-Language Modelscited by 44
- 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 44
- 2023CLAIR: Evaluating Image Captions with Large Language Modelscited by 43
- 2022Probing Classifiers are Unreliable for Concept Removal and Detectioncited by 40
- 2023Text encoders bottleneck compositionality in contrastive vision-language modelscited by 34
- 2025OmniVLA: An Omni-Modal Vision-Language-Action Model for Robot Navigationcited by 34
- 2024SFR-RAG: Towards Contextually Faithful LLMscited by 19
- 2025AVTrustBench: Assessing and Enhancing Reliability and Robustness in Audio-Visual LLMscited by 19
- 2025AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Modelscited by 18
- 2024MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluationcited by 17
- 2024OmnixR: Evaluating Omni-modality Language Models on Reasoning across Modalitiescited by 13
- 2024Is Your Large Language Model Knowledgeable or a Choices-Only Cheater?cited by 12
- 2024Why do LLaVA Vision-Language Models Reply to Images in English?cited by 11
- 2025RoboEgo System Card: An Omnimodal Model with Native Full Duplexitycited by 5
- 2025Learning Interpretable Features in Audio Latent Spaces via Sparse Autoencoderscited by 5
Built on by
Citing papers per year: 2026: 9.
9 papers.
- 2026When Vision Speaks for Soundcited by 2
- 2026Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videoscited by 2
- 2026Omni-DeepSearch: A Benchmark for Audio-Driven Omni-Modal Deep Searchcited by 1
- 2026From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMscited by 1
- 2026C3PO: Evaluating Cross-Modal Composition and Counterfactual Performance in Omnimodal Modelscited by 1
- 2026Omni-Streaming Thinkingcited by 1
- 2026Chain of Modality: From Static Fusion to Dynamic Orchestration in Omni-MLLMscited by 0
- 2026Seeing What Should Be Heard: Diagnosing and Repairing Cross-Modal Shortcuts in Omni-Modal LLMscited by 0
- 2026Devils in Question Relay: Source-Conditioned Relay Steering to Mitigate Hallucinations in Audio-visual Large Language Modelscited by 0
- 2026Devils in Question Relay: Source-Conditioned Relay Steering to Mitigate Hallucinations in Audio-visual Large Language Models29 Sep 2026
- 2026Seeing What Should Be Heard: Diagnosing and Repairing Cross-Modal Shortcuts in Omni-Modal LLMs29 Sep 2026
- 2026Omni-Streaming Thinking14 Sep 2026
- 2026C3PO: Evaluating Cross-Modal Composition and Counterfactual Performance in Omnimodal Models5 Aug 2026
- 2026Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videos17 Jul 2026
- 2026From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMs8 Jun 2026
- 2026When Vision Speaks for Sound13 May 2026
- 2026Omni-DeepSearch: A Benchmark for Audio-Driven Omni-Modal Deep Search9 May 2026
- 2026Chain of Modality: From Static Fusion to Dynamic Orchestration in Omni-MLLMs16 Apr 2026
Similar, not cited
Close in wording, no citation either way
- 2026Probing Cross-modal Information Hubs in Audio-Visual LLMs
- 2025Fork-Merge Decoding: Enhancing Multimodal Understanding in Audio-Visual Large Language Models
- 2026AVI-Bench: Toward Human-like Audio-Visual Intelligence of Omni-MLLMs
- 2025Bridging Ears and Eyes: Analyzing Audio and Visual Large Language Models to Humans in Visible Sound Recognition and Reducing Their Sensory Gap via Cross-Modal Distillation
- 2024AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?
- 2023Audio-Visual LLM for Video Understanding
- 2025Nexus: An Omni-Perceptive And -Interactive Model for Language, Audio, And Vision
- 2024Meerkat: Audio-Visual Large Language Model for Grounding in Space and Time