mlsanityData to 14 Oct 2026

Dr. SHAP-AV: Decoding Relative Modality Contributions via Shapley Attribution in Audio-Visual Speech Recognition

arXiv 2603.12046 ↗First posted 12 Mar 2026eess.AS cs.CV cs.SDcited by 1

Not on the map. Why?

Abstract (arXiv)

Audio-Visual Speech Recognition (AVSR) leverages both acoustic and visual information for robust recognition under noise. However, how models balance these modalities remains unclear. We present Dr. SHAP-AV, a framework using Shapley values to analyze modality contributions in AVSR. Through experiments on six models across two benchmarks and varying SNR levels, we introduce three analyses: Global SHAP for overall modality balance, Generative SHAP for contribution dynamics during decoding, and Temporal Alignment SHAP for input-output correspondence. Our findings reveal that models shift toward visual reliance under noise yet maintain high audio contributions even under severe degradation. Modality balance evolves during generation, temporal alignment holds under noise, and SNR is the dominant factor driving modality weighting. These findings expose a persistent audio bias, motivating ad-hoc modality-weighting mechanisms and Shapley-based attribution as a standard AVSR diagnostic.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 49 of its 59 references that are in the map. First posted on arXiv 12 Mar 2026.

Builds on

49 of 59 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  6. 2025Qwen3-VL Technical Reportcited by 2,793
  7. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  8. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 425
  9. 2018Deep Audio-Visual Speech Recognitioncited by 322
  10. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 308
  11. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
  12. 2022Matryoshka Representation Learningcited by 282
37 more
  1. 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 264
  2. 2016Lip Reading Sentences in the Wildcited by 228
  3. 2020Explaining by Removing: A Unified Framework for Model Explanationcited by 218
  4. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 207
  5. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 140
  6. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 139
  7. 2023End-to-End Speech Recognition: A Surveycited by 129
  8. 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 114
  9. 2018The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselinescited by 92
  10. 2024Matryoshka Multimodal Modelscited by 84
  11. 2023Multimodal Representation Learning by Alternating Unimodal Adaptationcited by 75
  12. 2018End-to-end Audiovisual Speech Recognitioncited by 73
  13. 2018Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecturecited by 64
  14. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languagescited by 64
  15. 2022Jointly Learning Visual and Auditory Speech Representations from Raw Datacited by 47
  16. 2022MM-SHAP: A Performance-agnostic Metric for Measuring Multimodal Contributions in Vision and Language Models & Taskscited by 43
  17. 2023Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoringcited by 42
  18. 2024Large Language Models are Strong Audio-Visual Speech Recognition Learnerscited by 42
  19. 2015Deep Multimodal Learning for Audio-Visual Speech Recognitioncited by 38
  20. 2024Whisper-Flamingo: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translationcited by 36
  21. 2025ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editingcited by 34
  22. 2024Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processingcited by 30
  23. 2023Explaining Reinforcement Learning with Shapley Valuescited by 26
  24. 2022Visual Context-driven Audio Feature Enhancement for Robust End-to-End Audio-Visual Speech Recognitioncited by 24
  25. 2025MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokenscited by 19
  26. 2024Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputscited by 15
  27. 2024A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognitioncited by 12
  28. 2025Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMscited by 11
  29. 2025Mitigating Attention Sinks and Massive Activations in Audio-Visual Speech Recognition with LLMscited by 11
  30. 2025Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approachcited by 10
  31. 2025Watch and Listen: Understanding Audio-Visual-Speech Moments with Multimodal LLMcited by 7
  32. 2024Uncovering the Visual Contribution in Audio-Visual Speech Recognitioncited by 6
  33. 2025MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognitioncited by 4
  34. 2025MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognitioncited by 4
  35. 2025Omni-AVSR: Towards Unified Multimodal Speech Recognition with Large Language Modelscited by 4
  36. 2025Interpreting the Role of Visemes in Audio-Visual Speech Recognitioncited by 2
  37. 2025Investigating Modality Contribution in Audio LLMs for Musiccited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026VIB-AVSR: Variational Information Bottleneck for Noise-Robust LLM-Based Audio-Visual Speech Recognitioncited by 0