# Conditional Image Generation by Conditioning Variational Auto-Encoders

arXiv: https://arxiv.org/abs/2102.12037 · HTML: https://mlsanity.com/paper/2102.12037/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 72 references that are in the map.
- First posted on arXiv 24 Feb 2021.

## Builds on (most cited first, 30 of 37)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [Likelihood Ratios for Out-of-Distribution Detection](https://mlsanity.com/paper/1906.02845/) (2019)
- [Do Deep Generative Models Know What They Don't Know?](https://mlsanity.com/paper/1810.09136/) (2018)
- [Ladder Variational Autoencoders](https://mlsanity.com/paper/1602.02282/) (2016)
- [Neural Processes](https://mlsanity.com/paper/1807.01622/) (2018)
- [Toward Multimodal Image-to-Image Translation](https://mlsanity.com/paper/1711.11586/) (2017)
- [Attentive Neural Processes](https://mlsanity.com/paper/1901.05761/) (2019)
- [Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images](https://mlsanity.com/paper/2011.10650/) (2020)
- [DRAW: A Recurrent Neural Network For Image Generation](https://mlsanity.com/paper/1502.04623/) (2015)
- [Generalization and Equilibrium in Generative Adversarial Nets (GANs)](https://mlsanity.com/paper/1703.00573/) (2017)
- [Pluralistic Image Completion](https://mlsanity.com/paper/1903.04227/) (2019)
- [Semantic Image Inpainting with Deep Generative Models](https://mlsanity.com/paper/1607.07539/) (2016)
- [Likelihood Regret: An Out-of-Distribution Detection Score For Variational Auto-encoder](https://mlsanity.com/paper/2003.02977/) (2020)
- [Recurrent Feature Reasoning for Image Inpainting](https://mlsanity.com/paper/2008.03737/) (2020)
- [EDDI: Efficient Dynamic Discovery of High-Value Information with Partial VAE](https://mlsanity.com/paper/1809.11142/) (2018)
- [Variational Autoencoder with Arbitrary Conditioning](https://mlsanity.com/paper/1806.02382/) (2018)
- [SPG-Net: Segmentation Prediction and Guidance Network for Image Inpainting](https://mlsanity.com/paper/1805.03356/) (2018)
- [Hierarchical VAEs Know What They Don't Know](https://mlsanity.com/paper/2102.08248/) (2021)

## Built on by (most cited first, 25 of 25)

- [Dense Policy: Bidirectional Autoregressive Learning of Actions](https://mlsanity.com/paper/2503.13217/) (2025)
- [Implicit Image-to-Image Schrodinger Bridge for Image Restoration](https://mlsanity.com/paper/2403.06069/) (2024)
- [VTAE: Variational Transformer Autoencoder with Manifolds Learning](https://mlsanity.com/paper/2304.00948/) (2023)
- [Generative AI for Autonomous Driving: Frontiers and Opportunities](https://mlsanity.com/paper/2505.08854/) (2025)
- [Matching aggregate posteriors in the variational autoencoder](https://mlsanity.com/paper/2311.07693/) (2023)
- [High Fidelity Image Synthesis With Deep VAEs In Latent Space](https://mlsanity.com/paper/2303.13714/) (2023)
- [Humanoid World Models: Open World Foundation Models for Humanoid Robotics](https://mlsanity.com/paper/2506.01182/) (2025)
- [ET-Former: Efficient Triplane Deformable Attention for 3D Semantic Scene Completion From Monocular Camera](https://mlsanity.com/paper/2410.11019/) (2024)
- [Probabilistic Shape Completion by Estimating Canonical Factors with Hierarchical VAE](https://mlsanity.com/paper/2212.03370/) (2022)
- [Fairness in Machine Learning-based Hand Load Estimation: A Case Study on Load Carriage Tasks](https://mlsanity.com/paper/2504.05610/) (2025)
- [Explanations Go Linear: Post-hoc Explainability for Tabular Data with Interpretable Meta-Encoding](https://mlsanity.com/paper/2504.20667/) (2025)
- [Diverse super-resolution with pretrained deep hiererarchical VAEs](https://mlsanity.com/paper/2205.10347/) (2022)
- [Target-aware Variational Auto-encoders for Ligand Generation with Multimodal Protein Representation Learning](https://mlsanity.com/paper/2309.16685/) (2023)
- [StableSemantics: A Synthetic Language-Vision Dataset of Semantic Representations in Naturalistic Images](https://mlsanity.com/paper/2406.13735/) (2024)
- [Edge-based Denoising Image Compression](https://mlsanity.com/paper/2409.10978/) (2024)
- [VisDiff: SDF-Guided Polygon Generation for Visibility Reconstruction and Recognition](https://mlsanity.com/paper/2410.05530/) (2024)
- [RestoreGrad: Signal Restoration Using Conditional Denoising Diffusion Models with Jointly Learned Prior](https://mlsanity.com/paper/2502.13574/) (2025)
- [Counterfactual experience augmented off-policy reinforcement learning](https://mlsanity.com/paper/2503.13842/) (2025)
- [DiffuMural: Restoring Dunhuang Murals with Multi-scale Diffusion](https://mlsanity.com/paper/2504.09513/) (2025)
- [One-shot Conditional Sampling: MMD meets Nearest Neighbors](https://mlsanity.com/paper/2509.25507/) (2025)
- [Crystal Generation using the Fully Differentiable Pipeline and Latent Space Optimization](https://mlsanity.com/paper/2601.04606/) (2026)
- [A Survey on Semantic Communication for Vision: Categories, Frameworks, Enabling Techniques, and Applications](https://mlsanity.com/paper/2601.22202/) (2026)
- [SemaPop: Semantic-Persona Conditioned and Controllable Population Synthesis](https://mlsanity.com/paper/2602.11569/) (2026)
- [Debiasing Sequential Recommendation with Time-aware Inverse Propensity Scoring](https://mlsanity.com/paper/2603.04986/) (2026)
- [An Elastic Shape Variational Autoencoder for Skeleton Pose Trajectories](https://mlsanity.com/paper/2605.09231/) (2026)

## Abstract (arXiv)

We present a conditional variational auto-encoder (VAE) which, to avoid the substantial cost of training from scratch, uses an architecture and training objective capable of leveraging a foundation model in the form of a pretrained unconditional VAE. To train the conditional VAE, we only need to train an artifact to perform amortized inference over the unconditional VAE's latent variables given a conditioning input. We demonstrate our approach on tasks including image inpainting, for which it outperforms state-of-the-art GAN-based approaches at faithfully representing the inherent uncertainty. We conclude by describing a possible application of our inpainting model, in which it is used to perform Bayesian experimental design for the purpose of guiding a sensor.