# Generative Models of Visually Grounded Imagination

arXiv: https://arxiv.org/abs/1705.10762 · HTML: https://mlsanity.com/paper/1705.10762/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 May 2017.

## Built on by (most cited first, 30 of 44)

- [Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models](https://mlsanity.com/paper/1911.03393/) (2019)
- [Generalized Multimodal ELBO](https://mlsanity.com/paper/2105.02470/) (2021)
- [Multimodal Conditional Image Synthesis with Product-of-Experts GANs](https://mlsanity.com/paper/2112.05130/) (2021)
- [Unsupervised Learning of Compositional Energy Concepts](https://mlsanity.com/paper/2111.03042/) (2021)
- [A survey of multimodal deep generative models](https://mlsanity.com/paper/2207.02127/) (2022)
- [On the Limitations of Multimodal VAEs](https://mlsanity.com/paper/2110.04121/) (2021)
- [Compositional Sculpting of Iterative Generative Processes](https://mlsanity.com/paper/2309.16115/) (2023)
- [Efficient LLM Comparative Assessment: a Product of Experts Framework for Pairwise Comparisons](https://mlsanity.com/paper/2405.05894/) (2024)
- [Learning Multimodal VAEs through Mutual Supervision](https://mlsanity.com/paper/2106.12570/) (2021)
- [Reconstructing Nonlinear Dynamical Systems from Multi-Modal Time Series](https://mlsanity.com/paper/2111.02922/) (2021)
- [Compositional Image Decomposition with Diffusion Models](https://mlsanity.com/paper/2406.19298/) (2024)
- [Learning Conditional Variational Autoencoders with Missing Covariates](https://mlsanity.com/paper/2203.01218/) (2022)
- [Unsupervised physics-informed disentanglement of multimodal data for high-throughput scientific discovery](https://mlsanity.com/paper/2202.03242/) (2022)
- [Scalable Multi-Task Gaussian Processes with Neural Embedding of Coregionalization](https://mlsanity.com/paper/2109.09261/) (2021)
- [Generalised Gaussian Process Latent Variable Models (GPLVM) with Stochastic Variational Inference](https://mlsanity.com/paper/2202.12979/) (2022)
- [Discriminative Multimodal Learning via Conditional Priors in Generative Models](https://mlsanity.com/paper/2110.04616/) (2021)
- [Learning Sequential Latent Variable Models from Multimodal Time Series Data](https://mlsanity.com/paper/2204.10419/) (2022)
- [Learning multi-modal generative models with permutation-invariant encoders and tighter variational objectives](https://mlsanity.com/paper/2309.00380/) (2023)
- [Multimodal Variational Autoencoder: a Barycentric View](https://mlsanity.com/paper/2412.20487/) (2024)
- [Aggregation of Dependent Expert Distributions in Multimodal Variational Autoencoders](https://mlsanity.com/paper/2505.01134/) (2025)
- [Hellinger Multimodal Variational Autoencoders](https://mlsanity.com/paper/2601.06572/) (2026)
- [Multimodal Adversarially Learned Inference with Factorized Discriminators](https://mlsanity.com/paper/2112.10384/) (2021)
- [Language-free Compositional Action Generation via Decoupling Refinement](https://mlsanity.com/paper/2307.03538/) (2023)
- [Improving Variational Autoencoder Estimation from Incomplete Data with Mixture Variational Families](https://mlsanity.com/paper/2403.03069/) (2024)
- [A Markov Random Field Multi-Modal Variational AutoEncoder](https://mlsanity.com/paper/2408.09576/) (2024)
- [From Concrete to Abstract: A Multimodal Generative Approach to Abstract Concept Learning](https://mlsanity.com/paper/2410.02365/) (2024)
- [Multi-Component VAE with Gaussian Markov Random Field](https://mlsanity.com/paper/2507.12165/) (2025)
- [Dynamical modeling of nonlinear latent factors in multiscale neural activity with real-time inference](https://mlsanity.com/paper/2512.12462/) (2025)
- [Learn from Anywhere: Rethinking Generalized Zero-Shot Learning with Limited Supervision](https://mlsanity.com/paper/2107.04952/) (2021)
- [Improving Multimodal Joint Variational Autoencoders through Normalizing Flows and Correlation Analysis](https://mlsanity.com/paper/2305.11832/) (2023)

## Abstract (arXiv)

It is easy for people to imagine what a man with pink hair looks like, even if they have never seen such a person before. We call the ability to create images of novel semantic concepts visually grounded imagination. In this paper, we show how we can modify variational auto-encoders to perform this task. Our method uses a novel training objective, and a novel product-of-experts inference network, which can handle partially specified (abstract) concepts in a principled and efficient way. We also propose a set of easy-to-compute evaluation metrics that capture our intuitive notions of what it means to have good visual imagination, namely correctness, coverage, and compositionality (the 3 C's). Finally, we perform a detailed comparison of our method with two existing joint image-attribute VAE methods (the JMVAE method of Suzuki et.al. and the BiVCCA method of Wang et.al.) by applying them to two datasets: the MNIST-with-attributes dataset (which we introduce here), and the CelebA dataset.