# Tensorizing flows: a tool for variational inference

arXiv: https://arxiv.org/abs/2305.02460 · HTML: https://mlsanity.com/paper/2305.02460/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 65 references that are in the map.
- First posted on arXiv 3 May 2023.

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [NICE: Non-linear Independent Components Estimation](https://mlsanity.com/paper/1410.8516/) (2014)
- [FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models](https://mlsanity.com/paper/1810.01367/) (2018)
- [Semi-Supervised Learning with Deep Generative Models](https://mlsanity.com/paper/1406.5298/) (2014)
- [Invertible Residual Networks](https://mlsanity.com/paper/1811.00995/) (2018)
- [Residual Flows for Invertible Generative Modeling](https://mlsanity.com/paper/1906.02735/) (2019)
- [Flow-based generative models for Markov chain Monte Carlo in lattice field theory](https://mlsanity.com/paper/1904.12072/) (2019)
- [Neural Variational Inference and Learning in Belief Networks](https://mlsanity.com/paper/1402.0030/) (2014)
- [The Born Supremacy: Quantum Advantage and Training of an Ising Born Machine](https://mlsanity.com/paper/1904.02214/) (2019)
- [Supervised Learning with Quantum-Inspired Tensor Networks](https://mlsanity.com/paper/1605.05775/) (2016)
- [Tensor-Train Density Estimation](https://mlsanity.com/paper/2108.00089/) (2021)
- [Computing Committor Functions for the Study of Rare Events Using Deep Learning](https://mlsanity.com/paper/1906.06285/) (2019)
- [VIREL: A Variational Inference Framework for Reinforcement Learning](https://mlsanity.com/paper/1811.01132/) (2018)
- [Meta-Amortized Variational Inference and Learning](https://mlsanity.com/paper/1902.01950/) (2019)
- [Generative modeling via tensor train sketching](https://mlsanity.com/paper/2202.11788/) (2022)
- [Deep composition of tensor-trains using squared inverse Rosenblatt transports](https://mlsanity.com/paper/2007.06968/) (2020)
- [A Modern Analysis of Hutchinson's Trace Estimator](https://mlsanity.com/paper/2012.12895/) (2020)
- [Learning Model Reparametrizations: Implicit Variational Inference by Fitting MCMC distributions](https://mlsanity.com/paper/1708.01529/) (2017)
- [Tensor Train Factorization and Completion under Noisy Data with Prior Analysis and Rank Estimation](https://mlsanity.com/paper/2010.06564/) (2020)
- [Variational Inference with NoFAS: Normalizing Flow with Adaptive Surrogate for Computationally Expensive Models](https://mlsanity.com/paper/2108.12657/) (2021)
- [Copula-Based Normalizing Flows](https://mlsanity.com/paper/2107.07352/) (2021)
- [Tensor-Train Networks for Learning Predictive Modeling of Multidimensional Data](https://mlsanity.com/paper/2101.09184/) (2021)

## Built on by (most cited first, 1 of 1)

- [A Review of Change of Variable Formulas for Generative Modeling](https://mlsanity.com/paper/2308.02652/) (2023)

## Abstract (arXiv)

Fueled by the expressive power of deep neural networks, normalizing flows have achieved spectacular success in generative modeling, or learning to draw new samples from a distribution given a finite dataset of training samples. Normalizing flows have also been applied successfully to variational inference, wherein one attempts to learn a sampler based on an expression for the log-likelihood or energy function of the distribution, rather than on data. In variational inference, the unimodality of the reference Gaussian distribution used within the normalizing flow can cause difficulties in learning multimodal distributions. We introduce an extension of normalizing flows in which the Gaussian reference is replaced with a reference distribution that is constructed via a tensor network, specifically a matrix product state or tensor train. We show that by combining flows with tensor networks on difficult variational inference tasks, we can improve on the results obtained by using either tool without the other.