mlsanityData to 14 Oct 2026

A Closer Look at Multimodal Representation Collapse

arXiv 2505.22483 ↗First posted 28 May 2025cs.LG cs.AI cs.CVcited by 28

Not on the map. Why?

Abstract (arXiv)

We aim to develop a fundamental understanding of modality collapse, a recently observed empirical phenomenon wherein models trained for multimodal fusion tend to rely only on a subset of the modalities, ignoring the rest. We show that modality collapse happens when noisy features from one modality are entangled, via a shared set of neurons in the fusion head, with predictive features from another, effectively masking out positive contributions from the predictive features of the former modality and leading to its collapse. We further prove that cross-modal knowledge distillation implicitly disentangles such representations by freeing up rank bottlenecks in the student encoder, denoising the fusion-head outputs without negatively impacting the predictive features from either modality. Based on the above findings, we propose an algorithm that prevents modality collapse through explicit basis reallocation, with applications in dealing with missing modalities. Extensive experiments on multiple multimodal benchmarks validate our theoretical claims. Project page: https://abhrac.github.io/mmcollapse/.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 38 of its 52 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

38 of 52 references.

References · 38

  1. 2022Generative Adversarial Networkscited by 5,627
  2. 2019Invariant Risk Minimizationcited by 1,854
  3. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  4. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  5. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  6. 2022Toy Models of Superpositioncited by 971
  7. 2019Contrastive Representation Distillationcited by 748
  8. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 645
  9. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  10. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 344
  11. 2023Multimodal Deep Learningcited by 258
  12. 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 245
26 more
  1. 2019Universal Approximation with Deep Narrow Networkscited by 200
  2. 2022Are Multimodal Transformers Robust to Missing Modality?cited by 186
  3. 2019Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Modelscited by 178
  4. 2023Multi-modal Learning with Missing Modality via Shared-Specific Feature Modellingcited by 163
  5. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 154
  6. 2018Handling Incomplete Heterogeneous Data using VAEscited by 149
  7. 2020Handling Missing Data with Graph Representation Learningcited by 133
  8. 2017Learning Independent Causal Mechanismscited by 105
  9. 2021Generalized Multimodal ELBOcited by 87
  10. 2018CentralNet: a Multilayer Approach for Multimodal Fusioncited by 86
  11. 2020Unpaired Multi-modal Segmentation via Knowledge Distillationcited by 81
  12. 2022Polysemanticity and Capacity in Neural Networkscited by 77
  13. 2022M3Care: Learning with Missing Modalities in Multimodal Healthcare Datacited by 72
  14. 2020VAEM: a Deep Generative Model for Heterogeneous Mixed Type Datacited by 66
  15. 2022Dynamic Multimodal Fusioncited by 55
  16. 2019Brain Tumor Segmentation on MRI with Missing Modalitiescited by 42
  17. 2022Geometric Multimodal Contrastive Representation Learningcited by 40
  18. 2022Mitigating Modality Collapse in Multimodal VAEs via Impartial Optimizationcited by 32
  19. 2021Properties from Mechanisms: An Equivariance Perspective on Identifiable Representation Learningcited by 30
  20. 2020Relating by Contrasting: A Data-efficient Framework for Multimodal Generative Modelscited by 27
  21. 2023Learning Missing Modal Electronic Health Records with Unified Multi-modal Data Embedding and Modality-Aware Attentioncited by 25
  22. 2022SGD and Weight Decay Secretly Minimize the Rank of Your Neural Networkcited by 19
  23. 2023What Causes Polysemanticity? An Alternative Origin Story of Mixed Selectivity from Incidental Causescited by 19
  24. 2024DeNetDM: Debiasing by Network Depth Modulationcited by 6
  25. 2022Cross-Modal Fusion Distillation for Fine-Grained Sketch-Based Image Retrievalcited by 4
  26. 2024Learning Conditional Invariances through Non-Commutativitycited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 23.

28 papers.

Sort:

By area:

  1. 2025The Achilles' Heel of LLMs: How Altering a Handful of Neurons Can Cripple Language Abilitiescited by 7
  2. 2025DeepfakeBench-MM: A Comprehensive Benchmark for Multimodal Deepfake Detectioncited by 3
  3. 2025Towards Relaxed Multimodal Inputs for Gait-based Parkinson's Disease Assessmentcited by 1
  4. 2025Countering Multi-modal Representation Collapse through Rank-targeted Fusioncited by 1
  5. 2026BARE: Towards Bias-Aware and Reasoning-Enhanced One-Tower Visual Groundingcited by 1
  6. 2026Diverse via bounded Agreement: Geometric Regularization for Multimodal Fusioncited by 1
  7. 2026Uncertainty-Aware Multimodal Learning via Conformal Shapley Intervalscited by 1
  8. 2026ModalImmune: Immunity Driven Unlearning via Self Destructive Trainingcited by 1
  9. 2026Feature-level Interaction Explanations in Multimodal Transformerscited by 1
  10. 2025Preserving Marker Specificity with Lightweight Channel-Independent Representation Learningcited by 0
  11. 2026Where is the multimodal goal post? On the Ability of Foundation Models to Recognize Highlights in Videoscited by 0
  12. 2026A Contrastive Variational AutoEncoder for NSCLC Survival Prediction with Missing Modalitiescited by 0
16 more
  1. 2026When One Modality Rules Them All: Backdoor Modality Collapse in Multimodal Diffusion Modelscited by 0
  2. 2026Towards Comprehensive Real-Time Scene Understanding in Ophthalmic Surgery through Multimodal Image Fusioncited by 0
  3. 2026Token-Efficient Multimodal Reasoning via Image Prompt Packagingcited by 0
  4. 2026MiMIC: Mitigating Visual Modality Collapse in Universal Multimodal Retrieval While Avoiding Semantic Misalignmentcited by 0
  5. 2026Learning Emergent Modular Representations in Multi-modality Medical Vision Foundation Modelscited by 0
  6. 2026LLM Features Can Hurt GNNs: Concatenation Interference on Homophilous Graph Benchmarkscited by 0
  7. 2026DynaWM: Dynamics-Aware Distillation with World Model and Momentum Targets for Smooth Locomotion over Continuous Stairscited by 0
  8. 2026General Incomplete Multimodal Learning via Dynamic Quality Perceptioncited by 0
  9. 2026Scaling Representation Diversity: Modulated Attention and Reconstructive Regularization for Visual Groundingcited by 0
  10. 2026Mitigating Strong-Modality Collapse in Multimodal Learning via Inverted Asymmetric Fusioncited by 0
  11. 2026More Perspectives, Stronger Signals: Multi-Perspective Enhancement and Progressive Fusion for Multimodal Entity Representation Learningcited by 0
  12. 2026Aligned but Flattened: Analyzing the Trade-off between Cultural Alignment and Diversity in LLMscited by 0
  13. 2026Characterizing Text Branch Sensitivity in Medical Vision-Language Segmentation via Evidence Decouplingcited by 0
  14. 2026OFBD: Object-Focused Background Debiasing for Long-Tailed Learningcited by 0
  15. 2026On the Geometry of Multimodal Saturation: Riemannian VICRegcited by 0
  16. 2026GeoPID: Decomposing and Steering Visual Information in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learning
  2. 2026Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness
  3. 2026Modality Collapse as Mismatched Decoding: Information-Theoretic Limits of Multimodal LLMs
  4. 2026Mitigating Multimodal LLMs Hallucinations via Relevance Propagation at Inference Time
  5. 2025Multi-modal Synthetic Data Training and Model Collapse: Insights from VLMs and Diffusion Models
  6. 2026Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition
  7. 2025VisiPruner: Decoding Discontinuous Cross-Modal Dynamics for Efficient Multimodal LLMs
  8. 2025Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Models