# Where to Diffuse, How to Diffuse, and How to Get Back: Automated Learning for Multivariate Diffusions

arXiv: https://arxiv.org/abs/2302.07261 · HTML: https://mlsanity.com/paper/2302.07261/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 31 of its 43 references that are in the map.
- First posted on arXiv 14 Feb 2023.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 31)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [Maximum Likelihood Training of Score-Based Diffusion Models](https://mlsanity.com/paper/2101.09258/) (2021)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [WaveGrad: Estimating Gradients for Waveform Generation](https://mlsanity.com/paper/2009.00713/) (2020)
- [A Continuous Time Framework for Discrete Denoising Models](https://mlsanity.com/paper/2205.14987/) (2022)
- [Augmented Neural ODEs](https://mlsanity.com/paper/1904.01681/) (2019)
- [Sliced Score Matching: A Scalable Approach to Density and Score Estimation](https://mlsanity.com/paper/1905.07088/) (2019)
- [Black Box Variational Inference](https://mlsanity.com/paper/1401.0118/) (2013)
- [Ladder Variational Autoencoders](https://mlsanity.com/paper/1602.02282/) (2016)
- [Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images](https://mlsanity.com/paper/2011.10650/) (2020)
- [Score-Based Generative Modeling with Critically-Damped Langevin Diffusion](https://mlsanity.com/paper/2112.07068/) (2021)
- [A Variational Perspective on Diffusion-Based Generative Models and Score Matching](https://mlsanity.com/paper/2106.02808/) (2021)
- [Symbolic Music Generation with Diffusion Models](https://mlsanity.com/paper/2103.16091/) (2021)
- [A Complete Recipe for Stochastic Gradient MCMC](https://mlsanity.com/paper/1506.04696/) (2015)
- [UNIT-DDPM: UNpaired Image Translation with Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2104.05358/) (2021)
- [BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling](https://mlsanity.com/paper/1902.02102/) (2019)
- [Hierarchical Variational Models](https://mlsanity.com/paper/1511.02386/) (2015)
- [Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models](https://mlsanity.com/paper/2002.07101/) (2020)

## Built on by (most cited first, 29 of 29)

- [SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers](https://mlsanity.com/paper/2401.08740/) (2024)
- [Protein Design with Guided Discrete Diffusion](https://mlsanity.com/paper/2305.20009/) (2023)
- [Stochastic interpolants with data-dependent couplings](https://mlsanity.com/paper/2310.03725/) (2023)
- [Variational Flow Matching for Graph Generation](https://mlsanity.com/paper/2406.04843/) (2024)
- [Neural Flow Diffusion Models: Learnable Forward Process for Improved Diffusion Modelling](https://mlsanity.com/paper/2404.12940/) (2024)
- [Heavy-Tailed Diffusion Models](https://mlsanity.com/paper/2410.14171/) (2024)
- [On the Challenges and Opportunities in Generative AI](https://mlsanity.com/paper/2403.00025/) (2024)
- [Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalities](https://mlsanity.com/paper/2411.01053/) (2024)
- [SDE Matching: Scalable and Simulation-Free Training of Latent Stochastic Differential Equations](https://mlsanity.com/paper/2502.02472/) (2025)
- [Neural Diffusion Models](https://mlsanity.com/paper/2310.08337/) (2023)
- [A Complete Recipe for Diffusion Generative Models](https://mlsanity.com/paper/2303.01748/) (2023)
- [DiffEnc: Variational Diffusion with a Learned Encoder](https://mlsanity.com/paper/2310.19789/) (2023)
- [Generative Fractional Diffusion Models](https://mlsanity.com/paper/2310.17638/) (2023)
- [Efficient Integrators for Diffusion Generative Models](https://mlsanity.com/paper/2310.07894/) (2023)
- [Structure-Guided Adversarial Training of Diffusion Models](https://mlsanity.com/paper/2402.17563/) (2024)
- [Variational Schr"odinger Diffusion Models](https://mlsanity.com/paper/2405.04795/) (2024)
- [What's the score? Automated Denoising Score Matching for Nonlinear Diffusions](https://mlsanity.com/paper/2407.07998/) (2024)
- [Rectified Noise: A Generative Model Using Positive-incentive Noise](https://mlsanity.com/paper/2511.07911/) (2025)
- [Whitened Score Diffusion: A Structured Prior for Imaging Inverse Problems](https://mlsanity.com/paper/2505.10311/) (2025)
- [Critically-Damped Higher-Order Langevin Dynamics for Generative Modeling](https://mlsanity.com/paper/2506.21741/) (2025)
- [Fractional Diffusion Bridge Models](https://mlsanity.com/paper/2511.01795/) (2025)
- [Free energy Estimation on Any State Space](https://mlsanity.com/paper/2605.31063/) (2026)
- [Flow Map Learning via Nongradient Vector Flow](https://mlsanity.com/paper/2607.26398/) (2026)
- [Learning variational autoencoders via MCMC speed measures](https://mlsanity.com/paper/2308.13731/) (2023)
- [Towards Fast Stochastic Sampling in Diffusion Generative Models](https://mlsanity.com/paper/2402.07211/) (2024)
- [Multidimensional Adaptive Coefficient for Inference Trajectory Optimization in Flow and Diffusion](https://mlsanity.com/paper/2404.14161/) (2024)
- [Weak Supervision Dynamic KL-Weighted Diffusion Models Guided by Large Language Models](https://mlsanity.com/paper/2502.00826/) (2025)
- [Learning Normalized Energy Models for Linear Inverse Problems](https://mlsanity.com/paper/2605.15487/) (2026)
- [Gaussian Flow Dynamics: Simulation-Free Neural SDE Learning Beyond One-Time Marginals](https://mlsanity.com/paper/2610.04390/) (2026)

## Abstract (arXiv)

Diffusion-based generative models (DBGMs) perturb data to a target noise distribution and reverse this process to generate samples. The choice of noising process, or inference diffusion process, affects both likelihoods and sample quality. For example, extending the inference process with auxiliary variables leads to improved sample quality. While there are many such multivariate diffusions to explore, each new one requires significant model-specific analysis, hindering rapid prototyping and evaluation. In this work, we study Multivariate Diffusion Models (MDMs). For any number of auxiliary variables, we provide a recipe for maximizing a lower-bound on the MDMs likelihood without requiring any model-specific analysis. We then demonstrate how to parameterize the diffusion for a specified target noise distribution; these two points together enable optimizing the inference diffusion process. Optimizing the diffusion expands easy experimentation from just a few well-known processes to an automatic search over all linear diffusions. To demonstrate these ideas, we introduce two new specific diffusions as well as learn a diffusion process on the MNIST, CIFAR10, and ImageNet32 datasets. We show learned MDMs match or surpass bits-per-dims (BPDs) relative to fixed choices of diffusions for a given dataset and model architecture.