Bridging Traditional Explainability Methods and Multimodal Multilingual Models: An XAI-Based Analysis
On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning
Abstract (arXiv)
Multimodal Large Language Models (MLLMs) effectively integrate text and audio to interpret context in complex interactive dialogues. However, the internal mechanisms by which heterogeneous modalities influence model behavior remain opaque. While Shapley Values (SV) provide a robust, model-agnostic framework for local explainability in text-based NLP, their extension to multimodal data is hindered by cross-channel dependencies, intricate dialogue structures, and the prohibitive computational complexity of dense audio representations. In this work, we formalize a multimodal extension of the Shapley Value framework, treating discrete text tokens and aligned audio segments as cooperative features. To ensure computational feasibility, we deploy a suite of efficient estimation strategies: exact SV computation for low-dimensional inputs and sampling-based approximations - including Monte Carlo permutations and stratified sampling with Neyman-optimal allocation - to minimize variance under constrained computational budgets. To resolve the granularity mismatch between modalities, we propose Spectrogram-Guided Phonetic Alignment (SGPA), a novel preprocessing method that maps high-frequency audio streams to interpretable, word-aligned segments. Our contribution is twofold: first, we provide an open-source, model-agnostic Python package and a companion GUI for the computation and interactive visualization of multimodal attributions. Second, we evaluate our framework using curated subsets of the VoiceBench and Infinity Instruct datasets across diverse multilingual scenarios. Our experimental results reveal that input modality is a primary driver of attribution volatility and demonstrate that standard syntactic importance proxies often fail to predict model attention in multimodal, cross-lingual contexts.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. First posted on arXiv 21 Apr 2026.
Builds on
0 of 0 references.
No references here yet.
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026mllm-shap: A Shapley Value Explainability Platform for Text-Audio Multimodal Large Language Models
- 2025Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs
- 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey
- 2025Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark
- 2026MoST: Mixing Speech and Text with Modality-Aware Mixture of Experts
- 2026What Did the MLLM Hear? Token-Level Spectro-Temporal Grounding for Audio MLLM Explainability
- 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling
- 2025MinMo: A Multimodal Large Language Model for Seamless Voice Interaction