# Outsourced diffusion sampling: Efficient posterior inference in latent spaces of generative models

arXiv: https://arxiv.org/abs/2502.06999 · HTML: https://mlsanity.com/paper/2502.06999/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 67 of its 95 references that are in the map.
- First posted on arXiv 10 Feb 2025.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 67)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Diffusion Posterior Sampling for General Noisy Inverse Problems](https://mlsanity.com/paper/2209.14687/) (2022)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2304.05977/) (2023)
- [Building Normalizing Flows with Stochastic Interpolants](https://mlsanity.com/paper/2209.15571/) (2022)
- [Improving and generalizing flow-based generative models with minibatch optimal transport](https://mlsanity.com/paper/2302.00482/) (2023)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models](https://mlsanity.com/paper/1810.01367/) (2018)
- [Solving Inverse Problems in Medical Imaging with Score-Based Generative Models](https://mlsanity.com/paper/2111.08005/) (2021)
- [Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation](https://mlsanity.com/paper/2106.04399/) (2021)
- [GFlowNet Foundations](https://mlsanity.com/paper/2111.09266/) (2021)
- [Multisample Flow Matching: Straightening Flows with Minibatch Couplings](https://mlsanity.com/paper/2304.14772/) (2023)
- [Scalable Gradients for Stochastic Differential Equations](https://mlsanity.com/paper/2001.01328/) (2020)

## Built on by (most cited first, 17 of 17)

- [Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Models](https://mlsanity.com/paper/2508.09968/) (2025)
- [Steering Generative Models with Experimental Data for Protein Fitness Optimization](https://mlsanity.com/paper/2505.15093/) (2025)
- [Adaptive Destruction Processes for Diffusion Samplers](https://mlsanity.com/paper/2506.01541/) (2025)
- [Data-to-Energy Stochastic Dynamics](https://mlsanity.com/paper/2509.26364/) (2025)
- [Reinforced sequential Monte Carlo for amortised sampling](https://mlsanity.com/paper/2510.11711/) (2025)
- [Posterior Inference in Latent Space for Scalable Constrained Black-box Optimization](https://mlsanity.com/paper/2507.00480/) (2025)
- [Variational Flow Maps: Make Some Noise for One-Step Conditional Generation](https://mlsanity.com/paper/2603.07276/) (2026)
- [Lagrangian Perturbation Diffusion Steering: Latent Reinforcement Learning for Generative Policies](https://mlsanity.com/paper/2606.01151/) (2026)
- [Discrete diffusion samplers and bridges: Off-policy algorithms and applications in latent spaces](https://mlsanity.com/paper/2602.05961/) (2026)
- [LatentFlow: A General Framework for Conditioning Stochastic Processes](https://mlsanity.com/paper/2607.12922/) (2026)
- [Na"ive PAINE: Lightweight Text-to-Image Generation Improvement with Prompt Evaluation](https://mlsanity.com/paper/2603.12506/) (2026)
- [Hierarchical Variational Policies for Reward-Guided Diffusion](https://mlsanity.com/paper/2605.21661/) (2026)
- [Can We Predict The Human Preference For Text-to-Image Content Prior To Generation And Is It Even Useful To Do So?](https://mlsanity.com/paper/2606.05478/) (2026)
- [Gene Expression-Informed Jointly Controlled Generative Modeling for Precision Molecular Design](https://mlsanity.com/paper/2607.11978/) (2026)
- [FestDPO: Few-step Generator Alignment with Direct Preference Optimization](https://mlsanity.com/paper/2609.34673/) (2026)
- [High-Dimensional Simulation-Based Inference in Latent Spaces](https://mlsanity.com/paper/2609.37381/) (2026)
- [Amortised Inference through One-Step Implicit Sampling](https://mlsanity.com/paper/2610.12679/) (2026)

## Abstract (arXiv)

Any well-behaved generative model over a variable x can be expressed as a deterministic transformation of an exogenous ('outsourced') Gaussian noise variable z: x=fθ(z). In such a model (eg, a VAE, GAN, or continuous-time flow-based model), sampling of the target variable x ~ pθ(x) is straightforward, but sampling from a posterior distribution of the form p(xmidy) propto pθ(x)r(x,y), where r is a constraint function depending on an auxiliary variable y, is generally intractable. We propose to amortize the cost of sampling from such posterior distributions with diffusion models that sample a distribution in the noise space (z). These diffusion samplers are trained by reinforcement learning algorithms to enforce that the transformed samples fθ(z) are distributed according to the posterior in the data space (x). For many models and constraints, the posterior in noise space is smoother than in data space, making it more suitable for amortized inference. Our method enables conditional sampling under unconditional GAN, (H)VAE, and flow-based priors, comparing favorably with other inference methods. We demonstrate the proposed outsourced diffusion sampling in several experiments with large pretrained prior models: conditional image generation, reinforcement learning with human feedback, and protein structure generation.