mlsanityData to 7 Oct 2026

Mitigating Modality Collapse in Multimodal VAEs via Impartial Optimization

arXiv 2206.04496 ↗First posted 9 Jun 2022cs.LGcited by 32

Not on the map. Why?

Abstract (arXiv)

A number of variational autoencoders (VAEs) have recently emerged with the aim of modeling multimodal data, e.g., to jointly model images and their corresponding captions. Still, multimodal VAEs tend to focus solely on a subset of the modalities, e.g., by fitting the image while neglecting the caption. We refer to this limitation as modality collapse. In this work, we argue that this effect is a consequence of conflicting gradients during multimodal VAE training. We show how to detect the sub-graphs in the computational graphs where gradients conflict (impartiality blocks), as well as how to leverage existing gradient-conflict solutions from multitask learning to mitigate modality collapse. That is, to ensure impartial optimization across modalities. We apply our training framework to several multimodal VAE models, losses and datasets from the literature, and empirically show that our framework significantly improves the reconstruction performance, conditional generation, and coherence of the latent space across modalities.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 25 of its 41 references that are in the map. First posted on arXiv 9 Jun 2022.

Builds on

25 of 41 references.

References · 25

  1. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  2. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  3. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  4. 2019On the Convergence of Adam and Beyondcited by 834
  5. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  6. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 809
  7. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  8. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  9. 2015Importance Weighted Autoencoderscited by 454
  10. 2018Multimodal Generative Models for Scalable Weakly-Supervised Learningcited by 200
  11. 2020Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropoutcited by 199
  12. 2019Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Modelscited by 177
13 more
  1. 2018Handling Incomplete Heterogeneous Data using VAEscited by 148
  2. 2019From Variational to Deterministic Autoencoderscited by 131
  3. 2017Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inferencecited by 97
  4. 2021Generalized Multimodal ELBOcited by 86
  5. 2018Tighter Variational Bounds are Not Necessarily Bettercited by 76
  6. 2020VAEM: a Deep Generative Model for Heterogeneous Mixed Type Datacited by 66
  7. 2020Multimodal Generative Learning Utilizing Jensen-Shannon-Divergencecited by 48
  8. 2019MultiNet++: Multi-Stream Feature Aggregation and Geometric Loss Strategy for Multi-Task Learningcited by 45
  9. 2018Doubly Reparameterized Gradient Estimators for Monte Carlo Objectivescited by 41
  10. 2019A Variational Auto-Encoder Model for Stochastic Point Processescited by 30
  11. 2020Relating by Contrasting: A Data-efficient Framework for Multimodal Generative Modelscited by 27
  12. 2020Automatic Differentiation Variational Inference with Mixturescited by 19
  13. 2021Medical data wrangling with sequential variational autoencoderscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 5; 2024: 4; 2025: 5; 2026: 15.

32 papers.

Sort:

By area:

  1. 2022Multimodal learning with graphscited by 54
  2. 2025A Closer Look at Multimodal Representation Collapsecited by 25
  3. 2023Multimodal Data Integration for Oncology in the Era of Deep Neural Networks: A Reviewcited by 20
  4. 2024AlignRec: Aligning and Training in Multimodal Recommendationscited by 20
  5. 2024M3-JEPA: Multimodal Alignment via Multi-gate MoE based on the Joint-Embedding Predictive Architecturecited by 12
  6. 2025Neurosymbolic Diffusion Modelscited by 12
  7. 2024Multimodal Data Integration for Precision Oncology: Challenges and Future Directionscited by 10
  8. 2023Multi-modal Latent Diffusioncited by 9
  9. 2022Transductive Decoupled Variational Inference for Few-Shot Classificationcited by 7
  10. 2022Cooperation in the Latent Space: The Benefits of Adding Mixture Components in Variational Autoencoderscited by 6
  11. 2023Multi-task View Synthesis with Neural Radiance Fieldscited by 5
  12. 2023Learning multi-modal generative models with permutation-invariant encoders and tighter variational objectivescited by 4
20 more
  1. 2024Enhance the Robustness of Text-Centric Multimodal Alignmentscited by 4
  2. 2026Hellinger Multimodal Variational Autoencoderscited by 2
  3. 2023Improving Multimodal Joint Variational Autoencoders through Normalizing Flows and Correlation Analysiscited by 1
  4. 2025COPA: Comparing the incomparable in multi-objective model evaluationcited by 1
  5. 2025Towards Relaxed Multimodal Inputs for Gait-based Parkinson's Disease Assessmentcited by 1
  6. 2026Modality Collapse as Mismatched Decoding: Information-Theoretic Limits of Multimodal LLMscited by 1
  7. 2026A Sobering Look at Tabular Data Generation via Probabilistic Circuitscited by 1
  8. 2025Robust Multi-Omics Integration from Incomplete Modalities Significantly Improves Prediction of Alzheimer's Diseasecited by 0
  9. 2026When One Modality Rules Them All: Backdoor Modality Collapse in Multimodal Diffusion Modelscited by 0
  10. 2026SiMO: Single-Modality-Operable Multimodal Collaborative Perceptioncited by 0
  11. 2026Prompts Without Evidence: How Neuroimaging Mentions Shift Clinical Vision-Language Model Predictionscited by 0
  12. 2026Token-Efficient Multimodal Reasoning via Image Prompt Packagingcited by 0
  13. 2026Information Router for Mitigating Modality Dominance in Vision-Language Modelscited by 0
  14. 2026MiMIC: Mitigating Visual Modality Collapse in Universal Multimodal Retrieval While Avoiding Semantic Misalignmentcited by 0
  15. 2026GraphPL: Leveraging GNN for Efficient and Robust Modalities Imputation in Patchwork Learningcited by 0
  16. 2026Learning Emergent Modular Representations in Multi-modality Medical Vision Foundation Modelscited by 0
  17. 2026H"older++: Improving the Quality-Coherence Trade-off in Multimodal VAEscited by 0
  18. 2026ShapKO: Shapley-Adaptive Modality Knockout for Robust Multimodal Learningcited by 0
  19. 2026Mitigating Strong-Modality Collapse in Multimodal Learning via Inverted Asymmetric Fusioncited by 0
  20. 2026Enabling Vision and Cross-Modal Learning for Multimodal Stroke Recurrence Prediction: An Interpretable Two-Step Frameworkcited by 0