mlsanityData to 7 Oct 2026

Generative Models of Visually Grounded Imagination

arXiv 1705.10762 ↗First posted 30 May 2017cs.LG cs.CV stat.MLcited by 44

Not on the map. Why?

Abstract (arXiv)

It is easy for people to imagine what a man with pink hair looks like, even if they have never seen such a person before. We call the ability to create images of novel semantic concepts visually grounded imagination. In this paper, we show how we can modify variational auto-encoders to perform this task. Our method uses a novel training objective, and a novel product-of-experts inference network, which can handle partially specified (abstract) concepts in a principled and efficient way. We also propose a set of easy-to-compute evaluation metrics that capture our intuitive notions of what it means to have good visual imagination, namely correctness, coverage, and compositionality (the 3 C's). Finally, we perform a detailed comparison of our method with two existing joint image-attribute VAE methods (the JMVAE method of Suzuki et.al. and the BiVCCA method of Wang et.al.) by applying them to two datasets: the MNIST-with-attributes dataset (which we introduce here), and the CelebA dataset.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 May 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 2; 2020: 0; 2021: 12; 2022: 8; 2023: 4; 2024: 9; 2025: 6; 2026: 3.

44 papers.

Sort:

By area:

  1. 2019Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Modelscited by 177
  2. 2021Generalized Multimodal ELBOcited by 86
  3. 2021Multimodal Conditional Image Synthesis with Product-of-Experts GANscited by 75
  4. 2021Unsupervised Learning of Compositional Energy Conceptscited by 71
  5. 2022A survey of multimodal deep generative modelscited by 47
  6. 2021On the Limitations of Multimodal VAEscited by 37
  7. 2023Compositional Sculpting of Iterative Generative Processescited by 26
  8. 2024Efficient LLM Comparative Assessment: a Product of Experts Framework for Pairwise Comparisonscited by 22
  9. 2021Learning Multimodal VAEs through Mutual Supervisioncited by 18
  10. 2021Reconstructing Nonlinear Dynamical Systems from Multi-Modal Time Seriescited by 16
  11. 2024Compositional Image Decomposition with Diffusion Modelscited by 12
  12. 2022Learning Conditional Variational Autoencoders with Missing Covariatescited by 10
32 more
  1. 2022Unsupervised physics-informed disentanglement of multimodal data for high-throughput scientific discoverycited by 6
  2. 2021Scalable Multi-Task Gaussian Processes with Neural Embedding of Coregionalizationcited by 5
  3. 2021Discriminative Multimodal Learning via Conditional Priors in Generative Modelscited by 4
  4. 2022Generalised Gaussian Process Latent Variable Models (GPLVM) with Stochastic Variational Inferencecited by 4
  5. 2022Learning Sequential Latent Variable Models from Multimodal Time Series Datacited by 4
  6. 2023Learning multi-modal generative models with permutation-invariant encoders and tighter variational objectivescited by 4
  7. 2024Multimodal Variational Autoencoder: a Barycentric Viewcited by 4
  8. 2025Aggregation of Dependent Expert Distributions in Multimodal Variational Autoencoderscited by 3
  9. 2021Multimodal Adversarially Learned Inference with Factorized Discriminatorscited by 2
  10. 2023Language-free Compositional Action Generation via Decoupling Refinementcited by 2
  11. 2024Improving Variational Autoencoder Estimation from Incomplete Data with Mixture Variational Familiescited by 2
  12. 2024A Markov Random Field Multi-Modal Variational AutoEncodercited by 2
  13. 2024From Concrete to Abstract: A Multimodal Generative Approach to Abstract Concept Learningcited by 2
  14. 2025Multi-Component VAE with Gaussian Markov Random Fieldcited by 2
  15. 2025Dynamical modeling of nonlinear latent factors in multiscale neural activity with real-time inferencecited by 2
  16. 2026Hellinger Multimodal Variational Autoencoderscited by 2
  17. 2021Learn from Anywhere: Rethinking Generalized Zero-Shot Learning with Limited Supervisioncited by 1
  18. 2023Improving Multimodal Joint Variational Autoencoders through Normalizing Flows and Correlation Analysiscited by 1
  19. 2025ShaLa: Multimodal Shared Latent Space Modellingcited by 1
  20. 2025Universal Multi-Domain Translation via Diffusion Routerscited by 1
  21. 2019Imitation Learning of Factored Multi-agent Reactive Modelscited by 0
  22. 2021Recommending Burgers based on Pizza Preferences: Addressing Data Sparsity with a Product of Expertscited by 0
  23. 2021Communication-Efficient Federated Linear and Deep Generalized Canonical Correlation Analysiscited by 0
  24. 2022A Graphical Model for Fusing Diverse Microbiome Datacited by 0
  25. 2022Generalizing Multimodal Variational Methods to Setscited by 0
  26. 2022OrthoGAN:High-Precision Image Generation for Teeth Orthodontic Visualizationcited by 0
  27. 2024Deep Latent Variable Modeling of Physiological Signalscited by 0
  28. 2024Improving Fine-Grained Control via Aggregation of Multiple Diffusion Modelscited by 0
  29. 2024Enhancing Unimodal Latent Representations in Multimodal VAEs through Iterative Amortized Inferencecited by 0
  30. 2025Disentangling Multiplex Spatial-Temporal Transition Graph Representation Learning for Socially Enhanced POI Recommendationcited by 0
  31. 2026Characterizing the Predictive Impact of Modalities with Supervised Latent-Variable Modelingcited by 0
  32. 2026CoVAE: correlated multimodal generative modelingcited by 0