# Reliable Multimodal Learning Via Multi-Level Adaptive DeConfusion

arXiv: https://arxiv.org/abs/2502.19674 · HTML: https://mlsanity.com/paper/2502.19674/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 36 of its 65 references that are in the map.
- First posted on arXiv 27 Feb 2025.

## Builds on (most cited first, 30 of 36)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [What Makes Training Multi-Modal Classification Networks Hard?](https://mlsanity.com/paper/1905.12681/) (2019)
- [SkipNet: Learning Dynamic Routing in Convolutional Networks](https://mlsanity.com/paper/1711.09485/) (2017)
- [BERT Loses Patience: Fast and Robust Inference with Early Exit](https://mlsanity.com/paper/2006.04152/) (2020)
- [An Empirical Study of Training End-to-End Vision-and-Language Transformers](https://mlsanity.com/paper/2111.02387/) (2021)
- [MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2005.03545/) (2020)
- [Gated Multimodal Units for Information Fusion](https://mlsanity.com/paper/1702.01992/) (2017)
- [EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition](https://mlsanity.com/paper/1908.08498/) (2019)
- [UniT: Multimodal Multitask Learning with a Unified Transformer](https://mlsanity.com/paper/2102.10772/) (2021)
- [Learning Robust Representations via Multi-View Information Bottleneck](https://mlsanity.com/paper/2002.07017/) (2020)
- [Adaptive Neural Networks for Efficient Inference](https://mlsanity.com/paper/1702.07811/) (2017)
- [Listen to Look: Action Recognition by Previewing Audio](https://mlsanity.com/paper/1912.04487/) (2019)
- [What Makes Multi-modal Learning Better than Single (Provably)](https://mlsanity.com/paper/2106.04538/) (2021)
- [Supervised Multimodal Bitransformers for Classifying Images and Text](https://mlsanity.com/paper/1909.02950/) (2019)
- [More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classification](https://mlsanity.com/paper/2008.05457/) (2020)
- [Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)](https://mlsanity.com/paper/2203.12221/) (2022)
- [PMR: Prototypical Modal Rebalance for Multimodal Learning](https://mlsanity.com/paper/2211.07089/) (2022)
- [Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks](https://mlsanity.com/paper/2202.05306/) (2022)
- [Trusted Multi-View Classification](https://mlsanity.com/paper/2102.02051/) (2021)
- [Provable Dynamic Fusion for Low-Quality Multimodal Data](https://mlsanity.com/paper/2306.02050/) (2023)
- [Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Features](https://mlsanity.com/paper/2210.06756/) (2022)
- [Decoding Brain Representations by Multimodal Learning of Neural Activity and Visual Features](https://mlsanity.com/paper/1810.10974/) (2018)
- [IDK Cascades: Fast Deep Learning by Learning not to Overthink](https://mlsanity.com/paper/1706.00885/) (2017)
- [Diagnosing and Re-learning for Balanced Multimodal Learning](https://mlsanity.com/paper/2407.09705/) (2024)
- [A Variational Information Bottleneck Approach to Multi-Omics Data Integration](https://mlsanity.com/paper/2102.03014/) (2021)
- [CubeMLP: An MLP-based Model for Multimodal Sentiment Analysis and Depression Estimation](https://mlsanity.com/paper/2207.14087/) (2022)
- [On-the-fly Modulation for Balanced Multimodal Learning](https://mlsanity.com/paper/2410.11582/) (2024)
- [Uncertainty-Aware Multi-View Representation Learning](https://mlsanity.com/paper/2201.05776/) (2022)

## Abstract (arXiv)

Multimodal learning enhances the performance of various machine learning tasks by leveraging complementary information across different modalities. However, existing methods often learn multimodal representations that retain substantial inter-class confusion, making it difficult to achieve high-confidence predictions, particularly in real-world scenarios with low-quality or noisy data. To address this challenge, we propose Multi-Level Adaptive DeConfusion (MLAD), which eliminates inter-class confusion in multimodal data at both global and sample levels, significantly enhancing the classification reliability of multimodal models. Specifically, MLAD first learns class-wise latent distributions with global-level confusion removed via dynamic-exit modality encoders that adapt to the varying discrimination difficulty of each class and a cross-class residual reconstruction mechanism. Subsequently, MLAD further removes sample-specific confusion through sample-adaptive cross-modality rectification guided by confusion-free modality priors. These priors are constructed from low-confusion modality features, identified by evaluating feature confusion using the learned class-wise latent distributions and selecting those with low confusion via a Gaussian mixture model. Experiments demonstrate that MLAD outperforms state-of-the-art methods across multiple benchmarks and exhibits superior reliability.