# Latent Normalizing Flows for Discrete Sequences

arXiv: https://arxiv.org/abs/1901.10548 · HTML: https://mlsanity.com/paper/1901.10548/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jan 2019.

## Built on by (most cited first, 30 of 67)

- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution](https://mlsanity.com/paper/2310.16834/) (2023)
- [Simplified and Generalized Masked Diffusion for Discrete Data](https://mlsanity.com/paper/2406.04329/) (2024)
- [Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions](https://mlsanity.com/paper/2102.05379/) (2021)
- [Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning](https://mlsanity.com/paper/2208.04202/) (2022)
- [Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models](https://mlsanity.com/paper/2103.04922/) (2021)
- [A Survey on Generative Diffusion Model](https://mlsanity.com/paper/2209.02646/) (2022)
- [Simple Guidance Mechanisms for Discrete Diffusion Models](https://mlsanity.com/paper/2412.10193/) (2024)
- [Concrete Score Matching: Generalized Score Matching for Discrete Data](https://mlsanity.com/paper/2211.00802/) (2022)
- [Energy-Based Diffusion Language Models for Text Generation](https://mlsanity.com/paper/2410.21357/) (2024)
- [Bayesian Flow Networks](https://mlsanity.com/paper/2308.07037/) (2023)
- [Copula Flows for Synthetic Data Generation](https://mlsanity.com/paper/2101.00598/) (2021)
- [Continuous Diffusion Model for Language Modeling](https://mlsanity.com/paper/2502.11564/) (2025)
- [Universal approximation property of invertible neural networks](https://mlsanity.com/paper/2204.07415/) (2022)
- [Controllable Text Generation via Probability Density Estimation in the Latent Space](https://mlsanity.com/paper/2212.08307/) (2022)
- [TACTiS-2: Better, Faster, Simpler Attentional Copulas for Multivariate Time Series](https://mlsanity.com/paper/2310.01327/) (2023)
- [CaRiNG: Learning Temporal Causal Representation under Non-Invertible Generation Process](https://mlsanity.com/paper/2401.14535/) (2024)
- [Latent Bayesian Optimization via Autoregressive Normalizing Flows](https://mlsanity.com/paper/2504.14889/) (2025)
- [On the Universality of Volume-Preserving and Coupling-Based Normalizing Flows](https://mlsanity.com/paper/2402.06578/) (2024)
- [Sparse Probabilistic Circuits via Pruning and Growing](https://mlsanity.com/paper/2211.12551/) (2022)
- [CtrlDiff: Boosting Large Diffusion Language Models with Dynamic Block Prediction and Controllable Generation](https://mlsanity.com/paper/2505.14455/) (2025)
- [Personalized Public Policy Analysis in Social Sciences using Causal-Graphical Normalizing Flows](https://mlsanity.com/paper/2202.03281/) (2022)
- [Semi-Discrete Normalizing Flows through Differentiable Tessellation](https://mlsanity.com/paper/2203.06832/) (2022)
- [PassFlow: Guessing Passwords with Generative Flows](https://mlsanity.com/paper/2105.06165/) (2021)
- [Conditional Independence Testing via Latent Representation Learning](https://mlsanity.com/paper/2209.01547/) (2022)
- [Scaling Probabilistic Circuits via Monarch Matrices](https://mlsanity.com/paper/2506.12383/) (2025)
- [Low-Rank Constraints for Fast Inference in Structured Models](https://mlsanity.com/paper/2201.02715/) (2022)
- [TO-FLOW: Efficient Continuous Normalizing Flows with Temporal Optimization adjoint with Moving Speed](https://mlsanity.com/paper/2203.10335/) (2022)
- [Probabilistic Integral Circuits](https://mlsanity.com/paper/2310.16986/) (2023)
- [Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows](https://mlsanity.com/paper/2507.00425/) (2025)

## Abstract (arXiv)

Normalizing flows are a powerful class of generative models for continuous random variables, showing both strong model flexibility and the potential for non-autoregressive generation. These benefits are also desired when modeling discrete random variables such as text, but directly applying normalizing flows to discrete sequences poses significant additional challenges. We propose a VAE-based generative model which jointly learns a normalizing flow-based distribution in the latent space and a stochastic mapping to an observed discrete space. In this setting, we find that it is crucial for the flow-based distribution to be highly multimodal. To capture this property, we propose several normalizing flow architectures to maximize model flexibility. Experiments consider common discrete sequence tasks of character-level language modeling and polyphonic music generation. Our results indicate that an autoregressive flow-based model can match the performance of a comparable autoregressive baseline, and a non-autoregressive flow-based model can improve generation speed with a penalty to performance.