# A Tale of Two Latent Flows: Learning Latent Space Normalizing Flow with Short-run Langevin Flow for Approximate Inference

arXiv: https://arxiv.org/abs/2301.09300 · HTML: https://mlsanity.com/paper/2301.09300/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 44 references that are in the map.
- First posted on arXiv 23 Jan 2023.

## Builds on (most cited first, 30 of 32)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [NICE: Non-linear Independent Components Estimation](https://mlsanity.com/paper/1410.8516/) (2014)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows](https://mlsanity.com/paper/1906.12320/) (2019)
- [GraphAF: a Flow-based Autoregressive Model for Molecular Graph Generation](https://mlsanity.com/paper/2001.09382/) (2020)
- [Variational Lossy Autoencoder](https://mlsanity.com/paper/1611.02731/) (2016)
- [A Theory of Generative ConvNet](https://mlsanity.com/paper/1602.03264/) (2016)
- [Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model](https://mlsanity.com/paper/1904.09770/) (2019)
- [Efficient GAN-Based Anomaly Detection](https://mlsanity.com/paper/1802.06222/) (2018)
- [Learning Latent Space Energy-Based Prior Model](https://mlsanity.com/paper/2006.08205/) (2020)
- [Flow Contrastive Estimation of Energy-Based Models](https://mlsanity.com/paper/1912.00589/) (2019)
- [VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation](https://mlsanity.com/paper/1903.01434/) (2019)
- [Cooperative Training of Descriptor and Generator Networks](https://mlsanity.com/paper/1609.09408/) (2016)
- [Trajectory Prediction with Latent Belief Energy-Based Model](https://mlsanity.com/paper/2104.03086/) (2021)
- [Learning Generative Vision Transformer with Energy-Based Latent Space for Saliency Prediction](https://mlsanity.com/paper/2112.13528/) (2021)
- [Maximum Entropy Generators for Energy-Based Models](https://mlsanity.com/paper/1901.08508/) (2019)
- [WaveFlow: A Compact Flow-based Model for Raw Audio](https://mlsanity.com/paper/1912.01219/) (2019)
- [A Tale of Two Flows: Cooperative Learning of Langevin Flow and Normalizing Flow Toward Energy-Based Model](https://mlsanity.com/paper/2205.06924/) (2022)
- [Learning Feature-to-Feature Translator by Alternating Back-Propagation for Generative Zero-Shot Learning](https://mlsanity.com/paper/1904.10056/) (2019)
- [Joint Training of Variational Auto-Encoder and Latent Energy-Based Model](https://mlsanity.com/paper/2006.06059/) (2020)
- [Learning Noise-Aware Encoder-Decoder from Noisy Labels by Alternating Back-Propagation for Saliency Detection](https://mlsanity.com/paper/2007.12211/) (2020)
- [Learning Dynamic Generator Model by Alternating Back-Propagation Through Time](https://mlsanity.com/paper/1812.10587/) (2018)
- [MAE: Mutual Posterior-Divergence Regularization for Variational AutoEncoders](https://mlsanity.com/paper/1901.01498/) (2019)
- [Latent Space Energy-Based Model of Symbol-Vector Coupling for Text Generation and Classification](https://mlsanity.com/paper/2108.11556/) (2021)
- [Motion-Based Generator Model: Unsupervised Disentanglement of Appearance, Trackable and Intrackable Motions in Dynamic Patterns](https://mlsanity.com/paper/1911.11294/) (2019)

## Built on by (most cited first, 7 of 7)

- [Latent Plan Transformer for Trajectory Abstraction: Planning as Latent Space Inference](https://mlsanity.com/paper/2402.04647/) (2024)
- [Molecule Design by Latent Prompt Transformer](https://mlsanity.com/paper/2310.03253/) (2023)
- [Likelihood-Based Generative Radiance Field with Latent Space Energy-Based Model for 3D-Aware Disentangled Image Representation](https://mlsanity.com/paper/2304.07918/) (2023)
- [Learning Energy-based Variational Latent Prior for VAEs](https://mlsanity.com/paper/2510.00260/) (2025)
- [Learning Multimodal Latent Space with EBM Prior and MCMC Inference](https://mlsanity.com/paper/2408.10467/) (2024)
- [STANLEY: Stochastic Gradient Anisotropic Langevin Dynamics for Learning Energy-Based Models](https://mlsanity.com/paper/2310.12667/) (2023)
- [Molecule Design by Latent Prompt Transformer](https://mlsanity.com/paper/2402.17179/) (2024)

## Abstract (arXiv)

We study a normalizing flow in the latent space of a top-down generator model, in which the normalizing flow model plays the role of the informative prior model of the generator. We propose to jointly learn the latent space normalizing flow prior model and the top-down generator model by a Markov chain Monte Carlo (MCMC)-based maximum likelihood algorithm, where a short-run Langevin sampling from the intractable posterior distribution is performed to infer the latent variables for each observed example, so that the parameters of the normalizing flow prior and the generator can be updated with the inferred latent variables. We show that, under the scenario of non-convergent short-run MCMC, the finite step Langevin dynamics is a flow-like approximate inference model and the learning objective actually follows the perturbation of the maximum likelihood estimation (MLE). We further point out that the learning framework seeks to (i) match the latent space normalizing flow and the aggregated posterior produced by the short-run Langevin flow, and (ii) bias the model from MLE such that the short-run Langevin flow inference is close to the true posterior. Empirical results of extensive experiments validate the effectiveness of the proposed latent space normalizing flow model in the tasks of image generation, image reconstruction, anomaly detection, supervised image inpainting and unsupervised image recovery.