# Robust Multimodal Learning via Representation Decoupling

arXiv: https://arxiv.org/abs/2407.04458 · HTML: https://mlsanity.com/paper/2407.04458/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 44 references that are in the map.
- First posted on arXiv 5 Jul 2024.

## Builds on (most cited first, 29 of 29)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [GhostNet: More Features from Cheap Operations](https://mlsanity.com/paper/1911.11907/) (2019)
- [Balanced Multimodal Learning via On-the-fly Gradient Modulation](https://mlsanity.com/paper/2203.15332/) (2022)
- [Probabilistic Face Embeddings](https://mlsanity.com/paper/1904.09658/) (2019)
- [Multi-modal Learning with Missing Modality via Shared-Specific Feature Modelling](https://mlsanity.com/paper/2307.14126/) (2023)
- [Probabilistic Embeddings for Cross-Modal Retrieval](https://mlsanity.com/paper/2101.05068/) (2021)
- [Data Uncertainty Learning in Face Recognition](https://mlsanity.com/paper/2003.11339/) (2020)
- [ACNet: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentation](https://mlsanity.com/paper/1905.10089/) (2019)
- [HeMIS: Hetero-Modal Image Segmentation](https://mlsanity.com/paper/1607.05194/) (2016)
- [Word Representations via Gaussian Embedding](https://mlsanity.com/paper/1412.6623/) (2014)
- [Efficient RGB-D Semantic Segmentation for Indoor Scene Analysis](https://mlsanity.com/paper/2011.06961/) (2020)
- [Modality Distillation with Multiple Stream Networks for Action Recognition](https://mlsanity.com/paper/1806.07110/) (2018)
- [ShapeConv: Shape-aware Convolutional Layer for Indoor RGB-D Semantic Segmentation](https://mlsanity.com/paper/2108.10528/) (2021)
- [RGB-D Salient Object Detection: A Survey](https://mlsanity.com/paper/2008.00230/) (2020)
- [D3D: Distilled 3D Networks for Video Action Recognition](https://mlsanity.com/paper/1812.08249/) (2018)
- [Deep RGB-D Saliency Detection with Depth-Sensitive Attention and Automatic Multi-Modal Fusion](https://mlsanity.com/paper/2103.11832/) (2021)
- [A Dataset and Benchmark for Large-scale Multi-modal Face Anti-spoofing](https://mlsanity.com/paper/1812.00408/) (2018)
- [View-Invariant Probabilistic Embedding for Human Pose](https://mlsanity.com/paper/1912.01001/) (2019)
- [Multimodal Remote Sensing Benchmark Datasets for Land Cover Classification with A Shared and Specific Feature Learning Model](https://mlsanity.com/paper/2105.10196/) (2021)
- [MMANet: Margin-aware Distillation and Modality-aware Regularization for Incomplete Multimodal Learning](https://mlsanity.com/paper/2304.08028/) (2023)
- [Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space](https://mlsanity.com/paper/1504.06654/) (2015)
- [Learning with privileged information via adversarial discriminative modality distillation](https://mlsanity.com/paper/1810.08437/) (2018)
- [Probabilistic Compositional Embeddings for Multimodal Image Retrieval](https://mlsanity.com/paper/2204.05845/) (2022)
- [SFANet: A Spectrum-aware Feature Augmentation Network for Visible-Infrared Person Re-Identification](https://mlsanity.com/paper/2102.12137/) (2021)
- [Brain tumor segmentation with missing modalities via latent multi-source correlation representation](https://mlsanity.com/paper/2003.08870/) (2020)
- [Integrating cross-modality hallucinated MRI with CT to aid mediastinal lung tumor segmentation](https://mlsanity.com/paper/1909.04542/) (2019)
- [Latent Heterogeneous Graph Network for Incomplete Multi-View Learning](https://mlsanity.com/paper/2208.13669/) (2022)

## Built on by (most cited first, 17 of 17)

- [DeMo: Decoupled Feature-Based Mixture of Experts for Multi-Modal Object Re-Identification](https://mlsanity.com/paper/2412.10650/) (2024)
- [Improving Multimodal Learning via Imbalanced Learning](https://mlsanity.com/paper/2507.10203/) (2025)
- [Boosting Multimodal Learning via Disentangled Gradient Learning](https://mlsanity.com/paper/2507.10213/) (2025)
- [Unbiased Dynamic Multimodal Fusion](https://mlsanity.com/paper/2603.19681/) (2026)
- [CyIN: Cyclic Informative Latent Space for Bridging Complete and Incomplete Multimodal Learning](https://mlsanity.com/paper/2602.04920/) (2026)
- [City-VLM: Towards Multidomain Perception Scene Understanding via Multimodal Incomplete Learning](https://mlsanity.com/paper/2507.12795/) (2025)
- [Complementarity-Supervised Spectral-Band Routing for Multimodal Emotion Recognition](https://mlsanity.com/paper/2603.13340/) (2026)
- [Modality Equilibrium Matters: Minor-Modality-Aware Adaptive Alternating for Cross-Modal Memory Enhancement](https://mlsanity.com/paper/2506.00030/) (2025)
- [Layer-Specific Lipschitz Modulation for Fault-Tolerant Multimodal Representation Learning](https://mlsanity.com/paper/2603.25103/) (2026)
- [Bridging MRI and PET physiology: Untangling complementarity through orthogonal representations](https://mlsanity.com/paper/2604.07154/) (2026)
- [Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibration](https://mlsanity.com/paper/2605.03820/) (2026)
- [Disentangled Fine-Grained Prototype Learning for Incomplete Image-Tabular Classification](https://mlsanity.com/paper/2606.05455/) (2026)
- [C3-Bench: A Context-Aware Change Captioning Benchmark](https://mlsanity.com/paper/2606.25445/) (2026)
- [Residual-Guided Expert Specialization for Incomplete Multimodal Learning](https://mlsanity.com/paper/2606.30355/) (2026)
- [General Incomplete Multimodal Learning via Dynamic Quality Perception](https://mlsanity.com/paper/2607.06943/) (2026)
- [UMSS: Towards Unsupervised Multi-modal Semantic Segmentation](https://mlsanity.com/paper/2607.12372/) (2026)
- [MIDAS: Mutual Information Disentanglement with Uncertainty-Aware Fusion for Incomplete Multimodal Sentiment Analysis](https://mlsanity.com/paper/2608.09986/) (2026)

## Abstract (arXiv)

Multimodal learning robust to missing modality has attracted increasing attention due to its practicality. Existing methods tend to address it by learning a common subspace representation for different modality combinations. However, we reveal that they are sub-optimal due to their implicit constraint on intra-class representation. Specifically, the sample with different modalities within the same class will be forced to learn representations in the same direction. This hinders the model from capturing modality-specific information, resulting in insufficient learning. To this end, we propose a novel Decoupled Multimodal Representation Network (DMRNet) to assist robust multimodal learning. Specifically, DMRNet models the input from different modality combinations as a probabilistic distribution instead of a fixed point in the latent space, and samples embeddings from the distribution for the prediction module to calculate the task loss. As a result, the direction constraint from the loss minimization is blocked by the sampled representation. This relaxes the constraint on the inference representation and enables the model to capture the specific information for different modality combinations. Furthermore, we introduce a hard combination regularizer to prevent DMRNet from unbalanced training by guiding it to pay more attention to hard modality combinations. Finally, extensive experiments on multimodal classification and segmentation tasks demonstrate that the proposed DMRNet outperforms the state-of-the-art significantly.