# Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs

arXiv: https://arxiv.org/abs/2511.22826 · HTML: https://mlsanity.com/paper/2511.22826/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- First posted on arXiv 28 Nov 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Built on by (most cited first, 7 of 7)

- [Instruction Anchor: Dissecting the Mechanistic Dynamics of Modality Arbitration](https://mlsanity.com/paper/2602.03677/) (2026)
- [Probing Cross-modal Information Hubs in Audio-Visual LLMs](https://mlsanity.com/paper/2605.10815/) (2026)
- [C3PO: Evaluating Cross-Modal Composition and Counterfactual Performance in Omnimodal Models](https://mlsanity.com/paper/2608.05381/) (2026)
- [Token-Efficient Multimodal Reasoning via Image Prompt Packaging](https://mlsanity.com/paper/2604.02492/) (2026)
- [A Systematic Study of Cross-Modal Typographic Attacks on Audio-Visual Reasoning](https://mlsanity.com/paper/2604.03995/) (2026)
- [Tri-PvP: Exposing Modality Bias in Omni-Modal Large Language Models through Perceptual-Propositional Evidence Conflicts](https://mlsanity.com/paper/2609.06011/) (2026)
- [Seeing What Should Be Heard: Diagnosing and Repairing Cross-Modal Shortcuts in Omni-Modal LLMs](https://mlsanity.com/paper/2609.36798/) (2026)

## Abstract (arXiv)

Despite remarkable advancements in Multimodal Large Language Models (MLLMs), a fundamental question remains: are MLLMs robust to contradicting modalities? To rigorously study this, we introduce MMA-Bench comprising videos and tasks that probe a model's reliance on specific modalities. Using black-box and white-box interpretability techniques, we provide a critical analysis of the brittleness of both open- and closed-sourced MLLMs. We show that current MLLMs struggle under misaligned audio-visual pairs and simple misleading text, thereby lacking robust multi-modal reasoning. Building on these findings, we propose a modality alignment tuning strategy to teach the model when to prioritize, leverage, or ignore specific modality cues. Through extensive experiments and analysis, we show that our alignment tuning yields demonstrably stronger multimodal grounding. This work provides both interpretability tools and a clear path toward developing MLLMs with intrinsically reliable cross-modal reasoning. Code and dataset will be publicly available.