mlsanityData to 14 Oct 2026

On the Limitations of Multimodal VAEs

arXiv 2110.04121 ↗First posted 8 Oct 2021cs.LGcited by 38

Not on the map. Why?

Abstract (arXiv)

Multimodal variational autoencoders (VAEs) have shown promise as efficient generative models for weakly-supervised data. Yet, despite their advantage of weak supervision, they exhibit a gap in generative quality compared to unimodal VAEs, which are completely unsupervised. In an attempt to explain this gap, we uncover a fundamental limitation that applies to a large family of mixture-based multimodal VAEs. We prove that the sub-sampling of modalities enforces an undesirable upper bound on the multimodal ELBO and thereby limits the generative quality of the respective models. Empirically, we showcase the generative quality gap on both synthetic and real data and present the tradeoffs between different variants of multimodal VAEs. We find that none of the existing approaches fulfills all desired criteria of an effective multimodal generative model when applied on more complex datasets than those used in previous benchmarks. In summary, we identify, formalize, and validate fundamental limitations of VAE-based approaches for modeling weakly-supervised data and discuss implications for real-world applications.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Builds on 30 of its 46 references that are in the map. First posted on arXiv 8 Oct 2021.

Builds on

30 of 46 references.

References · 30

  1. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  2. 2013Auto-Encoding Variational Bayescited by 4,053
  3. 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,039
  4. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
  5. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 941
  6. 2016Deep Variational Information Bottleneckcited by 930
  7. 2018Multimodal Unsupervised Image-to-Image Translationcited by 795
  8. 2019On Variational Bounds of Mutual Informationcited by 559
  9. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 312
  10. 2020Weakly-Supervised Disentanglement Without Compromisescited by 239
  11. 2018Pros and Cons of GAN Evaluation Measurescited by 224
  12. 2019Few-Shot Unsupervised Image-to-Image Translationcited by 222
18 more
  1. 2018Multimodal Generative Models for Scalable Weakly-Supervised Learningcited by 200
  2. 2019Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Modelscited by 178
  3. 2018Learning Factorized Multimodal Representationscited by 171
  4. 2019DIVA: Domain Invariant Variational Autoencoderscited by 115
  5. 2021M6: A Chinese Multimodal Pretrainercited by 91
  6. 2016Joint Multimodal Learning with Deep Generative Modelscited by 87
  7. 2021Generalized Multimodal ELBOcited by 87
  8. 2019The Incomplete Rosetta Stone Problem: Identifiability Results for Multi-View Nonlinear ICAcited by 86
  9. 2019Hetero-Modal Variational Encoder-Decoder for Joint Modality Completion and Segmentationcited by 68
  10. 2020Multimodal Generative Learning Utilizing Jensen-Shannon-Divergencecited by 48
  11. 2017Generative Models of Visually Grounded Imaginationcited by 44
  12. 2018Doubly Reparameterized Gradient Estimators for Monte Carlo Objectivescited by 41
  13. 2021A Variational Information Bottleneck Approach to Multi-Omics Data Integrationcited by 40
  14. 2020Relating by Contrasting: A Data-efficient Framework for Multimodal Generative Modelscited by 27
  15. 2018Disentangling by Partitioning: A Representation Learning Framework for Multimodal Sensory Datacited by 19
  16. 2019Multimodal Generative Models for Compositional Representation Learningcited by 14
  17. 2018Multi-Source Neural Variational Inferencecited by 9
  18. 2020Inverse Learning of Symmetriescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 7; 2024: 11; 2025: 9; 2026: 10.

38 papers.

Sort:

By area:

  1. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 174
  2. 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Modelscited by 72
  3. 2024On the Challenges and Opportunities in Generative AIcited by 31
  4. 2024Unity by Diversity: Improved Representation Learning in Multimodal VAEscited by 14
  5. 2025A Probabilistic Approach to Wildfire Spread Prediction Using a Denoising Diffusion Surrogate Modelcited by 11
  6. 2023Multi-modal Variational Autoencoders for normative modelling across multiple imaging modalitiescited by 10
  7. 2023Multi-modal Latent Diffusioncited by 9
  8. 2025Artificial Intelligence and Generative Models for Materials Discovery -- A Reviewcited by 8
  9. 2023A multimodal dynamical variational autoencoder for audiovisual speech representation learningcited by 6
  10. 2024Text-to-Image Cross-Modal Generation: A Systematic Reviewcited by 5
  11. 2023Learning multi-modal generative models with permutation-invariant encoders and tighter variational objectivescited by 4
  12. 2025Disentanglement of Variations with Multimodal Generative Modelingcited by 4
26 more
  1. 2023Improving Normative Modeling for Multi-modal Neuroimaging Data using mixture-of-product-of-experts variational autoencoderscited by 3
  2. 2024Multimodal ELBO with Diffusion Decoderscited by 3
  3. 2025Aggregation of Dependent Expert Distributions in Multimodal Variational Autoencoderscited by 3
  4. 2025Generative Models From and For Sampling-Based MPC: A Bootstrapped Approach For Adaptive Contact-Rich Manipulationcited by 3
  5. 2026Hellinger Multimodal Variational Autoencoderscited by 3
  6. 2022Benchmarking Multimodal Variational Autoencoders: CdSprites+ Dataset and Toolkitcited by 2
  7. 2024A Markov Random Field Multi-Modal Variational AutoEncodercited by 2
  8. 2026MUNI: Multimodal Unified Latent Diffusion for Coherent Any-to-Any Generationcited by 2
  9. 2023Improving Multimodal Joint Variational Autoencoders through Normalizing Flows and Correlation Analysiscited by 1
  10. 2023Learning Representations without Compositional Assumptionscited by 1
  11. 2024Bayesian Unsupervised Disentanglement of Anatomy and Geometry for Deep Groupwise Image Registrationcited by 1
  12. 2024A tutorial on multi-view autoencoders using the multi-view-AE librarycited by 1
  13. 2025ShaLa: Multimodal Shared Latent Space Modellingcited by 1
  14. 2025Self-Aug: Query and Entropy Adaptive Decoding for Large Vision-Language Modelscited by 1
  15. 2026AMShortcut: An Inference- and Training-Efficient Inverse Design Model for Amorphous Materialscited by 1
  16. 2026Clearer Sight, Fewer Lies: Oriented Pickup Preference Optimization for Multimodal Hallucination Mitigationcited by 1
  17. 2024Enhancing Unimodal Latent Representations in Multimodal VAEs through Iterative Amortized Inferencecited by 0
  18. 2024Correlating Variational Autoencoders Natively For Multi-View Imputationcited by 0
  19. 2025Ground Reaction Force Estimation via Time-aware Knowledge Distillationcited by 0
  20. 2025No Data Wasted: A Semi-supervised Generative Model for Incomplete Multi-view Data Integration with Missing Labelscited by 0
  21. 2026DIS2: Disentanglement Meets Distillation with Classwise Attention for Robust Remote Sensing Segmentation under Missing Modalitiescited by 0
  22. 2026Characterizing the Predictive Impact of Modalities with Supervised Latent-Variable Modelingcited by 0
  23. 2026AMGenC: Generating Charge Balanced Amorphous Materialscited by 0
  24. 2026Learning Multimodal Energy-Based Model with Multimodal Variational Auto-Encoder via MCMC Revisioncited by 0
  25. 2026Selective Correlation Based Knowledge Distillation for Ground Reaction Force Estimationcited by 0
  26. 2026H"older++: Improving the Quality-Coherence Trade-off in Multimodal VAEscited by 0