# Multimodal Variational Autoencoders for Semi-Supervised Learning: In Defense of Product-of-Experts

arXiv: https://arxiv.org/abs/2101.07240 · HTML: https://mlsanity.com/paper/2101.07240/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts.
- Builds on 14 of its 28 references that are in the map.
- First posted on arXiv 18 Jan 2021.

## Placed in

- Mixture-of-experts / [Multimodal MoE models](https://mlsanity.com/moe/multimodal-moe-models/)

## Builds on (most cited first, 14 of 14)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Semi-Supervised Learning with Deep Generative Models](https://mlsanity.com/paper/1406.5298/) (2014)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [Multimodal Generative Models for Scalable Weakly-Supervised Learning](https://mlsanity.com/paper/1802.05335/) (2018)
- [Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models](https://mlsanity.com/paper/1911.03393/) (2019)
- [Joint Multimodal Learning with Deep Generative Models](https://mlsanity.com/paper/1611.01891/) (2016)
- [Probing the Need for Visual Context in Multimodal Machine Translation](https://mlsanity.com/paper/1903.08678/) (2019)
- [Deep Variational Canonical Correlation Analysis](https://mlsanity.com/paper/1610.03454/) (2016)
- [Multimodal Generative Models for Compositional Representation Learning](https://mlsanity.com/paper/1912.05075/) (2019)
- [Latent Translation: Crossing Modalities by Bridging Generative Models](https://mlsanity.com/paper/1902.08261/) (2019)
- [Visual Dialogue without Vision or Dialogue](https://mlsanity.com/paper/1812.06417/) (2018)

## Built on by (most cited first, 9 of 9)

- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Sketch-Guided Text-to-Image Diffusion Models](https://mlsanity.com/paper/2211.13752/) (2022)
- [Pretraining is All You Need for Image-to-Image Translation](https://mlsanity.com/paper/2205.12952/) (2022)
- [Multimodal Conditional Image Synthesis with Product-of-Experts GANs](https://mlsanity.com/paper/2112.05130/) (2021)
- [Efficient LLM Comparative Assessment: a Product of Experts Framework for Pairwise Comparisons](https://mlsanity.com/paper/2405.05894/) (2024)
- [PARASOL: Parametric Style Control for Diffusion Image Synthesis](https://mlsanity.com/paper/2303.06464/) (2023)
- [Benchmarking Multimodal Variational Autoencoders: CdSprites+ Dataset and Toolkit](https://mlsanity.com/paper/2209.03048/) (2022)
- [SynLeaF: A Dual-Stage Multimodal Fusion Framework for Synthetic Lethality Prediction Across Pan- and Single-Cancer Contexts](https://mlsanity.com/paper/2603.22369/) (2026)
- [Fused Bayesian Flow Networks for Dual-Target Molecular Design](https://mlsanity.com/paper/2608.01007/) (2026)

## Abstract (arXiv)

Multimodal generative models should be able to learn a meaningful latent representation that enables a coherent joint generation of all modalities (e.g., images and text). Many applications also require the ability to accurately sample modalities conditioned on observations of a subset of the modalities. Often not all modalities may be observed for all training data points, so semi-supervised learning should be possible. In this study, we propose a novel product-of-experts (PoE) based variational autoencoder that have these desired properties. We benchmark it against a mixture-of-experts (MoE) approach and an approach of combining the modalities with an additional encoder network. An empirical evaluation shows that the PoE based models can outperform the contrasted models. Our experiments support the intuition that PoE models are more suited for a conjunctive combination of modalities.