When One Modality Sabotages the Others: A Diagnostic Lens on Multimodal Reasoning
On the map: LLM reasoning → Multimodal reasoning
also in 1 more
Abstract (arXiv)
Despite rapid growth in multimodal large language models (MLLMs), their reasoning traces remain opaque: it is often unclear which modality drives a prediction, how conflicts are resolved, or when one stream dominates. In this paper, we introduce modality sabotage, a diagnostic failure mode in which a high-confidence unimodal error overrides other evidence and misleads the fused result. To analyze such dynamics, we propose a lightweight, model-agnostic evaluation layer that treats each modality as an agent, producing candidate labels and a brief self-assessment used for auditing. A simple fusion mechanism aggregates these outputs, exposing contributors (modalities supporting correct outcomes) and saboteurs (modalities that mislead). Applying our diagnostic layer in a case study on multimodal emotion recognition benchmarks with foundation models revealed systematic reliability profiles, providing insight into whether failures may arise from dataset artifacts or model limitations. More broadly, our framework offers a diagnostic scaffold for multimodal reasoning, supporting principled auditing of fusion dynamics and informing possible interventions.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 9 of its 35 references that are in the map. First posted on arXiv 4 Nov 2025.
Builds on
9 of 35 references.
References · 9
- 2023GPT-4 Technical Reportcited by 18,132
- 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
- 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
- 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 636
- 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
- 2019RUBi: Reducing Unimodal Biases in Visual Question Answeringcited by 183
- 2024Large Vision-Language Models as Emotion Recognizers in Context Awarenesscited by 12
- 2024MIPS at SemEval-2024 Task 3: Multimodal Emotion-Cause Pair Extraction in Conversations with Multimodal Language Modelscited by 9
- 2024UMETTS: A Unified Framework for Emotional Text-to-Speech Synthesis with Multimodal Promptscited by 4
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025Learning What to Attend First: Modality-Importance-Guided Reasoning for Reliable Multimodal Emotion Understanding
- 2025Can Large Multimodal Models Actively Recognize Faulty Inputs? A Systematic Evaluation Framework of Their Input Scrutiny Ability
- 2025Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning
- 2025Mitigating Modal Imbalance in Multimodal Reasoning
- 2025Benchmarking and Bridging Emotion Conflicts for Multimodal Emotion Reasoning
- 2026Modality Fault Lines: Structural Corruptions Reveal Fragile Omni-Modal Reasoning
- 2026C3PO: Evaluating Cross-Modal Composition and Counterfactual Performance in Omnimodal Models
- 2026Follow the Clues, Frame the Truth: Hybrid-evidential Deductive Reasoning in Open-Vocabulary Multimodal Emotion Recognition