# Latent Constraints: Learning to Generate Conditionally from Unconditional Generative Models

arXiv: https://arxiv.org/abs/1711.05772 · HTML: https://mlsanity.com/paper/1711.05772/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Nov 2017.

## Built on by (most cited first, 30 of 39)

- [Score-based Generative Modeling in Latent Space](https://mlsanity.com/paper/2106.05931/) (2021)
- [Diffusion models as plug-and-play priors](https://mlsanity.com/paper/2206.09012/) (2022)
- [Symbolic Music Generation with Diffusion Models](https://mlsanity.com/paper/2103.16091/) (2021)
- [The Expressive Power of Low-Rank Adaptation](https://mlsanity.com/paper/2310.17513/) (2023)
- [What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus](https://mlsanity.com/paper/2105.02732/) (2021)
- [On the Origins of Linear Representations in Large Language Models](https://mlsanity.com/paper/2403.03867/) (2024)
- [Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Models](https://mlsanity.com/paper/2508.09968/) (2025)
- [Learning Interpretable Concepts: Unifying Causal Representation Learning and Foundation Models](https://mlsanity.com/paper/2402.09236/) (2024)
- [Generative Visual Prompt: Unifying Distributional Control of Pre-Trained Generative Models](https://mlsanity.com/paper/2209.06970/) (2022)
- [CausalAF: Causal Autoregressive Flow for Safety-Critical Driving Scenario Generation](https://mlsanity.com/paper/2110.13939/) (2021)
- [Towards Practical Plug-and-Play Diffusion Models](https://mlsanity.com/paper/2212.05973/) (2022)
- [Training-Free Constrained Generation With Stable Diffusion Models](https://mlsanity.com/paper/2502.05625/) (2025)
- [Variational AutoEncoder for Reference based Image Super-Resolution](https://mlsanity.com/paper/2106.04090/) (2021)
- [Uncertainty Estimation Using Riemannian Model Dynamics for Offline Reinforcement Learning](https://mlsanity.com/paper/2102.11327/) (2021)
- [Live Music Models](https://mlsanity.com/paper/2508.04651/) (2025)
- [Pragmatic Image Compression for Human-in-the-Loop Decision-Making](https://mlsanity.com/paper/2108.04219/) (2021)
- [Improved Input Reprogramming for GAN Conditioning](https://mlsanity.com/paper/2201.02692/) (2022)
- [Zero-Shot Conditioning of Score-Based Diffusion Models by Neuro-Symbolic Constraints](https://mlsanity.com/paper/2308.16534/) (2023)
- [Amortized Synthesis of Constrained Configurations Using a Differentiable Surrogate](https://mlsanity.com/paper/2106.09019/) (2021)
- [Explaining, Evaluating and Enhancing Neural Networks' Learned Representations](https://mlsanity.com/paper/2202.09374/) (2022)
- [Posterior Matching for Arbitrary Conditioning](https://mlsanity.com/paper/2201.12414/) (2022)
- [Learning with Constraint Learning: New Perspective, Solution Strategy and Various Applications](https://mlsanity.com/paper/2307.15257/) (2023)
- [Revisiting GANs by Best-Response Constraint: Perspective, Methodology, and Application](https://mlsanity.com/paper/2205.10146/) (2022)
- [Hierarchical Optimization-Derived Learning](https://mlsanity.com/paper/2302.05587/) (2023)
- [Improving Variational Autoencoder Estimation from Incomplete Data with Mixture Variational Families](https://mlsanity.com/paper/2403.03069/) (2024)
- [Semantically Adversarial Scenario Generation with Explicit Knowledge Guidance](https://mlsanity.com/paper/2106.04066/) (2021)
- [Flat Latent Manifolds for Human-machine Co-creation of Music](https://mlsanity.com/paper/2202.12243/) (2022)
- [LaPlaSS: Latent Space Planning for Stochastic Systems](https://mlsanity.com/paper/2404.07063/) (2024)
- [Interpretable Representation Learning of Cardiac MRI via Attribute Regularization](https://mlsanity.com/paper/2406.08282/) (2024)
- [Correcting Noisy Multilabel Predictions: Modeling Label Noise through Latent Space Shifts](https://mlsanity.com/paper/2502.14281/) (2025)

## Abstract (arXiv)

Deep generative neural networks have proven effective at both conditional and unconditional modeling of complex data distributions. Conditional generation enables interactive control, but creating new controls often requires expensive retraining. In this paper, we develop a method to condition generation without retraining the model. By post-hoc learning latent constraints, value functions that identify regions in latent space that generate outputs with desired attributes, we can conditionally sample from these regions with gradient-based optimization or amortized actor functions. Combining attribute constraints with a universal "realism" constraint, which enforces similarity to the data distribution, we generate realistic conditional images from an unconditional variational autoencoder. Further, using gradient-based optimization, we demonstrate identity-preserving transformations that make the minimal adjustment in latent space to modify the attributes of an image. Finally, with discrete sequences of musical notes, we demonstrate zero-shot conditional generation, learning latent constraints in the absence of labeled data or a differentiable reward function. Code with dedicated cloud instance has been made publicly available (https://goo.gl/STGMGx).