# Multimodal Generative Models for Compositional Representation Learning

arXiv: https://arxiv.org/abs/1912.05075 · HTML: https://mlsanity.com/paper/1912.05075/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Dec 2019.

## Built on by (most cited first, 14 of 14)

- [Versatile Diffusion: Text, Images and Variations All in One Diffusion Model](https://mlsanity.com/paper/2211.08332/) (2022)
- [Generalized Multimodal ELBO](https://mlsanity.com/paper/2105.02470/) (2021)
- [Multimodal Alignment and Fusion: A Survey](https://mlsanity.com/paper/2411.17040/) (2024)
- [A survey of multimodal deep generative models](https://mlsanity.com/paper/2207.02127/) (2022)
- [On the Limitations of Multimodal VAEs](https://mlsanity.com/paper/2110.04121/) (2021)
- [Cross-Modal 3D Shape Generation and Manipulation](https://mlsanity.com/paper/2207.11795/) (2022)
- [Multimodal Variational Autoencoders for Semi-Supervised Learning: In Defense of Product-of-Experts](https://mlsanity.com/paper/2101.07240/) (2021)
- [FairyTailor: A Multimodal Generative Framework for Storytelling](https://mlsanity.com/paper/2108.04324/) (2021)
- [Learning multi-modal generative models with permutation-invariant encoders and tighter variational objectives](https://mlsanity.com/paper/2309.00380/) (2023)
- [Aggregation of Dependent Expert Distributions in Multimodal Variational Autoencoders](https://mlsanity.com/paper/2505.01134/) (2025)
- [What Drives Compositional Generalization in Visual Generative Models? The Importance of Continuous Training Objectives](https://mlsanity.com/paper/2510.03075/) (2025)
- [Multimodal Deep Generative Model for Semi-Supervised Learning under Class Imbalance](https://mlsanity.com/paper/2605.06289/) (2026)
- [Enhancing Unimodal Latent Representations in Multimodal VAEs through Iterative Amortized Inference](https://mlsanity.com/paper/2410.11403/) (2024)
- [Emotion Recognition in Multi-Speaker Conversations through Speaker Identification, Knowledge Distillation, and Hierarchical Fusion](https://mlsanity.com/paper/2511.13731/) (2025)

## Abstract (arXiv)

As deep neural networks become more adept at traditional tasks, many of the most exciting new challenges concern multimodality---observations that combine diverse types, such as image and text. In this paper, we introduce a family of multimodal deep generative models derived from variational bounds on the evidence (data marginal likelihood). As part of our derivation we find that many previous multimodal variational autoencoders used objectives that do not correctly bound the joint marginal likelihood across modalities. We further generalize our objective to work with several types of deep generative model (VAE, GAN, and flow-based), and allow use of different model types for different modalities. We benchmark our models across many image, label, and text datasets, and find that our multimodal VAEs excel with and without weak supervision. Additional improvements come from use of GAN image models with VAE language models. Finally, we investigate the effect of language on learned image representations through a variety of downstream tasks, such as compositionally, bounding box prediction, and visual relation prediction. We find evidence that these image representations are more abstract and compositional than equivalent representations learned from only visual data.