# Variational Inference with NoFAS: Normalizing Flow with Adaptive Surrogate for Computationally Expensive Models

arXiv: https://arxiv.org/abs/2108.12657 · HTML: https://mlsanity.com/paper/2108.12657/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 73 references that are in the map.
- First posted on arXiv 28 Aug 2021.

## Builds on (most cited first, 18 of 18)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Masked Autoregressive Flow for Density Estimation](https://mlsanity.com/paper/1705.07057/) (2017)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [Black Box Variational Inference](https://mlsanity.com/paper/1401.0118/) (2013)
- [MADE: Masked Autoencoder for Distribution Estimation](https://mlsanity.com/paper/1502.03509/) (2015)
- [Advances in Variational Inference](https://mlsanity.com/paper/1711.05597/) (2017)
- [Hierarchical Variational Models](https://mlsanity.com/paper/1511.02386/) (2015)
- [Deep UQ: Learning deep neural network surrogate models for high dimensional uncertainty quantification](https://mlsanity.com/paper/1802.00850/) (2018)
- [A Probabilistic Graphical Model Foundation for Enabling Predictive Digital Twins at Scale](https://mlsanity.com/paper/2012.05841/) (2020)
- [The Generalized Reparameterization Gradient](https://mlsanity.com/paper/1610.02287/) (2016)
- [Accelerating Asymptotically Exact MCMC for Computationally Intensive Models via Local Approximations](https://mlsanity.com/paper/1402.1694/) (2014)
- [Parallel local approximation MCMC for expensive models](https://mlsanity.com/paper/1607.02788/) (2016)

## Built on by (most cited first, 7 of 7)

- [L-HYDRA: Multi-Head Physics-Informed Neural Networks](https://mlsanity.com/paper/2301.02152/) (2023)
- [InVAErt networks: a data-driven framework for model synthesis and identifiability analysis](https://mlsanity.com/paper/2307.12586/) (2023)
- [AdaAnn: Adaptive Annealing Scheduler for Probability Density Approximation](https://mlsanity.com/paper/2202.00792/) (2022)
- [A Primer on Variational Inference for Physics-Informed Deep Generative Modelling](https://mlsanity.com/paper/2409.06560/) (2024)
- [Tensorizing flows: a tool for variational inference](https://mlsanity.com/paper/2305.02460/) (2023)
- [Normalizing Flow Regression for Bayesian Inference with Offline Likelihood Evaluations](https://mlsanity.com/paper/2504.11554/) (2025)
- [LINFA: a Python library for variational inference with normalizing flow and annealing](https://mlsanity.com/paper/2307.04675/) (2023)

## Abstract (arXiv)

Fast inference of numerical model parameters from data is an important prerequisite to generate predictive models for a wide range of applications. Use of sampling-based approaches such as Markov chain Monte Carlo may become intractable when each likelihood evaluation is computationally expensive. New approaches combining variational inference with normalizing flow are characterized by a computational cost that grows only linearly with the dimensionality of the latent variable space, and rely on gradient-based optimization instead of sampling, providing a more efficient approach for Bayesian inference about the model parameters. Moreover, the cost of frequently evaluating an expensive likelihood can be mitigated by replacing the true model with an offline trained surrogate model, such as neural networks. However, this approach might generate significant bias when the surrogate is insufficiently accurate around the posterior modes. To reduce the computational cost without sacrificing inferential accuracy, we propose Normalizing Flow with Adaptive Surrogate (NoFAS), an optimization strategy that alternatively updates the normalizing flow parameters and surrogate model parameters. We also propose an efficient sample weighting scheme for surrogate model training that preserves global accuracy while effectively capturing high posterior density regions. We demonstrate the inferential and computational superiority of NoFAS against various benchmarks, including cases where the underlying model lacks identifiability. The source code and numerical experiments used for this study are available at https://github.com/cedricwangyu/NoFAS.