# When Does Quality-Aware Multimodal Fusion Matter? A Leakage-Safe Diagnostic for Decision-Level Dependence

arXiv: https://arxiv.org/abs/2606.26473 · HTML: https://mlsanity.com/paper/2606.26473/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 25 references that are in the map.
- First posted on arXiv 25 Jun 2026.

## Builds on (most cited first, 16 of 16)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [MOMENT: A Family of Open Time-series Foundation Models](https://mlsanity.com/paper/2402.03885/) (2024)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [Trusted Multi-View Classification with Dynamic Evidential Fusion](https://mlsanity.com/paper/2204.11423/) (2022)
- [Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2109.00412/) (2021)
- [The INTERSPEECH 2021 Computational Paralinguistics Challenge: COVID-19 Cough, COVID-19 Speech, Escalation & Primates](https://mlsanity.com/paper/2102.13468/) (2021)
- [Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities](https://mlsanity.com/paper/2404.16456/) (2024)
- [COLD Fusion: Calibrated and Ordinal Latent Distribution Fusion for Uncertainty-Aware Multimodal Emotion Recognition](https://mlsanity.com/paper/2206.05833/) (2022)
- [UNO: Uncertainty-aware Noisy-Or Multimodal Fusion for Unanticipated Input Degradation](https://mlsanity.com/paper/1911.05611/) (2019)
- [Multi-level Fusion of Wav2vec 2.0 and BERT for Multimodal Emotion Recognition](https://mlsanity.com/paper/2207.04697/) (2022)
- [Are you sure? Analysing Uncertainty Quantification Approaches for Real-world Speech Emotion Recognition](https://mlsanity.com/paper/2407.01143/) (2024)
- [Quality-Aware Multimodal Biometric Recognition](https://mlsanity.com/paper/2112.05827/) (2021)
- [Multimodal Sentiment Analysis with Missing Modality: A Knowledge-Transfer Approach](https://mlsanity.com/paper/2401.10747/) (2023)

## Abstract (arXiv)

Many multimodal systems estimate the reliability of each modality and weight their contributions to the final prediction. However, it remains unclear whether these scores influence model decisions or merely correlate with performance. We propose a simple diagnostic to test whether reliability information is used during inference. After training, the model and inputs are fixed while reliability scores are permuted across test examples. If predictions depend on these scores, performance should degrade. Experiments on StressID for stress recognition and CMU-MOSEI for sentiment analysis show that permuting reliability scores leaves performance unchanged despite substantial potential gains from selecting the best modality per example. In positive controls where reliability signals identify the correct modality, the same frozen fusion rules yield significant improvements, indicating that reliability signals influence fused decisions only when they reliably predict unimodal correctness.