mlsanityData to 7 Oct 2026

mllm-shap: A Shapley Value Explainability Platform for Text-Audio Multimodal Large Language Models

arXiv 2606.07531 ↗First posted 21 Apr 2026cs.CL cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We introduce mllm-shap, an open-source Python framework designed to extend Shapley Value (SV) explainability from text-only Large Language Models to Multimodal LLMs (MLLMs) processing joint text and audio inputs. While text-based attribution is well-studied, mllm-shap addresses three critical challenges unique to the multimodal regime: (1) Modality-aware coalition masking, which manages the interleaved processing of discrete text tokens and dense audio encoder frames. (2) Multi-turn conversation tracking, utilizing per-token metadata to maintain role and modality context. (3) Phonetic alignment-based token grouping, a novel technique that reduces the coalition space by 10x to 50x, rendering SV estimation computationally feasible for long-form audio. The platform implements five SV estimation strategies, including a Complementary Contributions (CC) estimator with Neyman-optimal allocation that demonstrates superior convergence over standard Monte Carlo baselines. mllm-shap is provided as a pip-installable package featuring an interactive web-based GUI for granular attribution visualization. To our knowledge, this is the first publicly available framework providing a complete, reproducible pipeline for SV-based explainability in text-audio MLLMs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 11 of its 19 references that are in the map. First posted on arXiv 21 Apr 2026.

Builds on

11 of 19 references.

References · 11

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  3. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  6. 2019Attention is not Explanationcited by 916
  7. 2020Explaining by Removing: A Unified Framework for Model Explanationcited by 218
  8. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
  9. 2025Infinity Instruct: Scaling Instruction Selection and Synthesis to Enhance Language Modelscited by 49
  10. 2024TokenSHAP: Interpreting Large Language Models with Monte Carlo Shapley Value Estimationcited by 41
  11. 2023Approximating the Shapley Value without Marginal Contributionscited by 34

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Bridging Traditional Explainability Methods and Multimodal Multilingual Models: An XAI-Based Analysis
  2. 2025Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs
  3. 2026Alignment Is All You Need: Instruction-Free Training for General Audio-Language Models
  4. 2026What Did the MLLM Hear? Token-Level Spectro-Temporal Grounding for Audio MLLM Explainability
  5. 2024Model Composition for Multimodal Large Language Models
  6. 2025AHELM: A Holistic Evaluation of Audio-Language Models
  7. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
  8. 2026Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs