mlsanityData to 7 Oct 2026

Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models

arXiv 1911.03393 ↗First posted 8 Nov 2019stat.ML cs.LGcited by 177

On the map: Mixture-of-experts → Multimodal MoE models

Abstract (arXiv)

Learning generative models that span multiple data modalities, such as vision and language, is often motivated by the desire to learn more useful, generalisable representations that faithfully capture common underlying factors between the modalities. In this work, we characterise successful learning of such models as the fulfillment of four criteria: i) implicit latent decomposition into shared and private subspaces, ii) coherent joint generation over all modalities, iii) coherent cross-generation across individual modalities, and iv) improved model learning for individual modalities through multi-modal integration. Here, we propose a mixture-of-experts multimodal variational autoencoder (MMVAE) to learn generative models on different sets of modalities, including a challenging image-language dataset, and demonstrate its ability to satisfy all four criteria, both qualitatively and quantitatively.

Key facts. Cited by 177 papers in the map (data through 7 Oct 2026). Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts. Builds on 38 of its 58 references that are in the map. First posted on arXiv 8 Nov 2019.

Builds on

38 of 58 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2013Auto-Encoding Variational Bayescited by 3,980
  4. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,906
  5. 2016Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Networkcited by 1,898
  6. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  7. 2015Learning Transferable Features with Deep Adaptation Networkscited by 1,323
  8. 2019On the Convergence of Adam and Beyondcited by 834
  9. 2016The Mythos of Model Interpretabilitycited by 716
  10. 2017Unsupervised Image-to-Image Translation Networkscited by 602
  11. 2014Deep Domain Confusion: Maximizing for Domain Invariancecited by 597
  12. 2015Importance Weighted Autoencoderscited by 454
26 more
  1. 2016Unsupervised Domain Adaptation with Residual Transfer Networkscited by 340
  2. 2016Domain Separation Networkscited by 317
  3. 2017Toward Multimodal Image-to-Image Translationcited by 295
  4. 2019Few-Shot Unsupervised Image-to-Image Translationcited by 222
  5. 2018Multimodal Generative Models for Scalable Weakly-Supervised Learningcited by 200
  6. 2016Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networkscited by 192
  7. 2018Learning Factorized Multimodal Representationscited by 171
  8. 2014A Convolutional Neural Network for Modelling Sentencescited by 163
  9. 2016Variational Autoencoder for Deep Learning of Images, Labels and Captionscited by 137
  10. 2016Unsupervised Cross-Domain Image Generationcited by 125
  11. 2017Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inferencecited by 97
  12. 2015Attribute2Image: Conditional Image Generation from Visual Attributescited by 93
  13. 2016Joint Multimodal Learning with Deep Generative Modelscited by 87
  14. 2018Tighter Variational Bounds are Not Necessarily Bettercited by 76
  15. 2017Auto-Encoding Sequential Monte Carlocited by 64
  16. 2017CycleGAN, a Master of Steganographycited by 63
  17. 2017Towards Deeper Understanding of Variational Autoencoding Modelscited by 56
  18. 2016Generative Image Modeling using Style and Structure Adversarial Networkscited by 50
  19. 2016Deep Variational Canonical Correlation Analysiscited by 44
  20. 2017Generative Models of Visually Grounded Imaginationcited by 44
  21. 2018Doubly Reparameterized Gradient Estimators for Monte Carlo Objectivescited by 41
  22. 2017Reinterpreting Importance-Weighted Autoencoderscited by 36
  23. 2019Latent Translation: Crossing Modalities by Bridging Generative Modelscited by 9
  24. 2018FlipDial: A Generative Model for Two-Way Visual Dialoguecited by 6
  25. 2018Visual Dialogue without Vision or Dialoguecited by 6
  26. 2017Deep Matching Autoencoderscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 24; 2022: 33; 2023: 26; 2024: 35; 2025: 32; 2026: 27.

177 papers.

Sort:

By area:

  1. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  2. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 189
  3. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  4. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  5. 2022Disentangled Representation Learningcited by 111
  6. 2021Generalized Multimodal ELBOcited by 86
  7. 2024Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusioncited by 79
  8. 2021Multimodal Conditional Image Synthesis with Product-of-Experts GANscited by 75
  9. 2023Factorized Contrastive Learning: Going Beyond Multi-view Redundancycited by 73
  10. 2022Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Featurescited by 63
  11. 2023Identifiability Results for Multimodal Contrastive Learningcited by 58
  12. 2022A survey of multimodal deep generative modelscited by 47
38 more
  1. 2021A Variational Information Bottleneck Approach to Multi-Omics Data Integrationcited by 40
  2. 2022Geometric Multimodal Contrastive Representation Learningcited by 40
  3. 2021Trustworthy Multimodal Regression with Mixture of Normal-inverse Gamma Distributionscited by 39
  4. 2021On the Limitations of Multimodal VAEscited by 37
  5. 2023Multimodal Variational Auto-encoder based Audio-Visual Segmentationcited by 37
  6. 2023BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP for Generic Natural Visual Stimulus Decodingcited by 36
  7. 2022Mitigating Modality Collapse in Multimodal VAEs via Impartial Optimizationcited by 32
  8. 2024Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectivescited by 30
  9. 2021Unsupervised Domain Adaptation with Variational Approximation for Cardiac Segmentationcited by 28
  10. 2022Adaptive Mixture of Experts Learning for Generalizable Face Anti-Spoofingcited by 28
  11. 2022Cross-Modal 3D Shape Generation and Manipulationcited by 27
  12. 2025Occ-LLM: Enhancing Autonomous Driving with Occupancy-Based Large Language Modelscited by 27
  13. 2025A Closer Look at Multimodal Representation Collapsecited by 25
  14. 2022Representation Uncertainty in Self-Supervised Learning as Variational Inferencecited by 22
  15. 2022Facial Video-based Remote Physiological Measurement via Self-supervised Learningcited by 22
  16. 2021Multimodal VAE Active Inference Controllercited by 21
  17. 2021Learning Multimodal VAEs through Mutual Supervisioncited by 18
  18. 2022Redesigning Multi-Scale Neural Network for Crowd Countingcited by 17
  19. 2023MoCaE: Mixture of Calibrated Experts Significantly Improves Object Detectioncited by 17
  20. 2021Reconstructing Nonlinear Dynamical Systems from Multi-Modal Time Seriescited by 16
  21. 2021Variational Multi-Task Learning with Gumbel-Softmax Priorscited by 15
  22. 2021Gaussian Mixture Variational Autoencoder with Contrastive Learning for Multi-Label Classificationcited by 15
  23. 2022Integrating Multimodal Data for Joint Generative Modeling of Complex Dynamicscited by 13
  24. 2023Unified Brain MR-Ultrasound Synthesis using Multi-Modal Hierarchical Representationscited by 13
  25. 2024Unity by Diversity: Improved Representation Learning in Multimodal VAEscited by 13
  26. 2024Predictive, scalable and interpretable knowledge tracing on structured domainscited by 13
  27. 2022Multiple Importance Sampling ELBO and Deep Ensembles of Variational Approximationscited by 12
  28. 2025STiL: Semi-supervised Tabular-Image Learning for Comprehensive Task-Relevant Information Exploration in Multimodal Classificationcited by 12
  29. 2021Audio-to-Image Cross-Modal Generationcited by 11
  30. 2024Gradient-Guided Modality Decoupling for Missing-Modality Robustnesscited by 11
  31. 2021Lifelong Mixture of Variational Autoencoderscited by 10
  32. 2023Multi-modal Variational Autoencoders for normative modelling across multiple imaging modalitiescited by 10
  33. 2021Multimodal Variational Autoencoders for Semi-Supervised Learning: In Defense of Product-of-Expertscited by 9
  34. 2022Multimodal sensor fusion in the latent representation spacecited by 9
  35. 2023Multi-modal Latent Diffusioncited by 9
  36. 2023Multi-modal Gaussian Process Variational Autoencoders for Neural and Behavioral Datacited by 9
  37. 2023Joint fMRI Decoding and Encoding with Latent Embedding Alignmentcited by 8
  38. 2024Unified Cross-Modal Medical Image Synthesis with Hierarchical Mixture of Product-of-Expertscited by 8

50 most cited of 177 All 177 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Multimodal Variational Autoencoder: a Barycentric View
  2. 2025Exploiting Mixture-of-Experts Redundancy Unlocks Multimodal Generative Abilities
  3. 2023Score-Based Multimodal Autoencoder
  4. 2025CoVAE: Consistency Training of Variational Autoencoders
  5. 2025A Survey of Generative Categories and Techniques in Multimodal Generative Models
  6. 2023Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts
  7. 2026Symbiotic-MoE: Unlocking the Synergy between Generation and Understanding
  8. 2026When Structured Sparse Autoencoders Learn Consistent Concepts Across Modalities