# Embracing the chaos: analysis and diagnosis of numerical instability in variational flows

arXiv: https://arxiv.org/abs/2307.06957 · HTML: https://mlsanity.com/paper/2307.06957/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 57 references that are in the map.
- First posted on arXiv 12 Jul 2023.

## Builds on (most cited first, 25 of 25)

- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Neural Spline Flows](https://mlsanity.com/paper/1906.04032/) (2019)
- [Black Box Variational Inference](https://mlsanity.com/paper/1401.0118/) (2013)
- [On Neural Differential Equations](https://mlsanity.com/paper/2202.02435/) (2022)
- [The Reversible Residual Network: Backpropagation Without Storing Activations](https://mlsanity.com/paper/1707.04585/) (2017)
- [Automatic Differentiation Variational Inference](https://mlsanity.com/paper/1603.00788/) (2016)
- [Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODE](https://mlsanity.com/paper/2006.02493/) (2020)
- [Sylvester Normalizing Flows for Variational Inference](https://mlsanity.com/paper/1803.05649/) (2018)
- [Understanding and Mitigating Exploding Inverses in Invertible Neural Networks](https://mlsanity.com/paper/2006.09347/) (2020)
- [Reversible Architectures for Arbitrarily Deep Residual Neural Networks](https://mlsanity.com/paper/1709.03698/) (2017)
- [Smooth Normalizing Flows](https://mlsanity.com/paper/2110.00351/) (2021)
- [MALI: A memory efficient and reverse accurate integrator for Neural ODEs](https://mlsanity.com/paper/2102.04668/) (2021)
- [Hamiltonian Variational Auto-Encoder](https://mlsanity.com/paper/1805.11328/) (2018)
- [MCMC Variational Inference via Uncorrected Hamiltonian Annealing](https://mlsanity.com/paper/2107.04150/) (2021)
- [Monte Carlo Variational Auto-Encoders](https://mlsanity.com/paper/2106.15921/) (2021)
- [Differentiable Annealed Importance Sampling and the Perils of Gradient Noise](https://mlsanity.com/paper/2107.10211/) (2021)
- [The Expressive Power of a Class of Normalizing Flow Models](https://mlsanity.com/paper/2006.00392/) (2020)
- [iVPF: Numerical Invertible Volume Preserving Flow for Efficient Lossless Compression](https://mlsanity.com/paper/2103.16211/) (2021)
- [Langevin Diffusion Variational Inference](https://mlsanity.com/paper/2208.07743/) (2022)
- [Bayesian inference via sparse Hamiltonian flows](https://mlsanity.com/paper/2203.05723/) (2022)
- [MixFlows: principled variational inference via mixed flows](https://mlsanity.com/paper/2205.07475/) (2022)
- [Robust normalizing flows using Bernstein-type polynomials](https://mlsanity.com/paper/2102.03509/) (2021)

## Built on by (most cited first, 6 of 6)

- [FALCON: Few-step Accurate Likelihoods for Continuous Flows](https://mlsanity.com/paper/2512.09914/) (2025)
- [Efficient Regression-Based Training of Normalizing Flows for Boltzmann Generators](https://mlsanity.com/paper/2506.01158/) (2025)
- [Variational Learning of Gaussian Process Latent Variable Models through Stochastic Gradient Annealed Importance Sampling](https://mlsanity.com/paper/2408.06710/) (2024)
- [Reparameterized Variational Rejection Sampling](https://mlsanity.com/paper/2309.14612/) (2023)
- [Asymptotically exact variational flows via involutive MCMC kernels](https://mlsanity.com/paper/2506.02162/) (2025)
- [Generative Topological Networks](https://mlsanity.com/paper/2406.15152/) (2024)

## Abstract (arXiv)

In this paper, we investigate the impact of numerical instability on the reliability of sampling, density evaluation, and evidence lower bound (ELBO) estimation in variational flows. We first empirically demonstrate that common flows can exhibit a catastrophic accumulation of error: the numerical flow map deviates significantly from the exact map -- which affects sampling -- and the numerical inverse flow map does not accurately recover the initial input -- which affects density and ELBO computations. Surprisingly though, we find that results produced by flows are often accurate enough for applications despite the presence of serious numerical instability. In this work, we treat variational flows as dynamical systems, and leverage shadowing theory to elucidate this behavior via theoretical guarantees on the error of sampling, density evaluation, and ELBO estimation. Finally, we develop and empirically test a diagnostic procedure that can be used to validate results produced by numerically unstable flows in practice.