# Expected information gain estimation via density approximations: Sample allocation and dimension reduction

arXiv: https://arxiv.org/abs/2411.08390 · HTML: https://mlsanity.com/paper/2411.08390/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 86 references that are in the map.
- First posted on arXiv 13 Nov 2024.

## Builds on (most cited first, 30 of 35)

- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [The frontier of simulation-based inference](https://mlsanity.com/paper/1911.01429/) (2019)
- [Learning Likelihoods with Conditional Normalizing Flows](https://mlsanity.com/paper/1912.00042/) (2019)
- [Variational Bayesian Optimal Experimental Design](https://mlsanity.com/paper/1903.05480/) (2019)
- [On Nesting Monte Carlo Estimators](https://mlsanity.com/paper/1709.06181/) (2017)
- [Optimal experimental design: Formulations and computations](https://mlsanity.com/paper/2407.16212/) (2024)
- [Inference via low-dimensional couplings](https://mlsanity.com/paper/1703.06131/) (2017)
- [Conditional Sampling with Monotone GANs: from Generative Models to Likelihood-Free Inference](https://mlsanity.com/paper/2006.06755/) (2020)
- [On the representation and learning of monotone triangular transport maps](https://mlsanity.com/paper/2009.10303/) (2020)
- [Estimating Mutual Information for Discrete-Continuous Mixtures](https://mlsanity.com/paper/1709.06212/) (2017)
- [Efficient Bayesian Experimental Design for Implicit Models](https://mlsanity.com/paper/1810.09912/) (2018)
- [Featurized Density Ratio Estimation](https://mlsanity.com/paper/2107.02212/) (2021)
- [Likelihood-informed dimension reduction for nonlinear inverse problems](https://mlsanity.com/paper/1403.4680/) (2014)
- [Certified dimension reduction in nonlinear Bayesian inverse problems](https://mlsanity.com/paper/1807.03712/) (2018)
- [Optimal low-rank approximations of Bayesian linear inverse problems](https://mlsanity.com/paper/1407.3463/) (2014)
- [Goal-Oriented Optimal Design of Experiments for Large-Scale Bayesian Linear Inverse Problems](https://mlsanity.com/paper/1802.06517/) (2018)
- [On minimax density estimation via measure transport](https://mlsanity.com/paper/2207.10231/) (2022)
- [Large-scale Bayesian optimal experimental design with derivative-informed projected neural network](https://mlsanity.com/paper/2201.07925/) (2022)
- [Triangular Flows for Generative Modeling: Statistical Consistency, Smoothness Classes, and Fast Rates](https://mlsanity.com/paper/2112.15595/) (2021)
- [Multilevel Monte Carlo estimation of expected information gains](https://mlsanity.com/paper/1811.07546/) (2018)
- [Variational Bayesian Optimal Experimental Design with Normalizing Flows](https://mlsanity.com/paper/2404.13056/) (2024)
- [Deep composition of tensor-trains using squared inverse Rosenblatt transports](https://mlsanity.com/paper/2007.06968/) (2020)
- [Bayesian model calibration for block copolymer self-assembly: Likelihood-free inference and expected information gain computation via measure transport](https://mlsanity.com/paper/2206.11343/) (2022)
- [A layered multiple importance sampling scheme for focused optimal Bayesian experimental design](https://mlsanity.com/paper/1903.11187/) (2019)
- [Bayesian decision-theoretic design of experiments under an alternative model](https://mlsanity.com/paper/1909.12570/) (2019)
- [Data-Free Likelihood-Informed Dimension Reduction of Bayesian Inverse Problems](https://mlsanity.com/paper/2102.13245/) (2021)
- [Design Amortization for Bayesian Optimal Experimental Design](https://mlsanity.com/paper/2210.03283/) (2022)

## Built on by (most cited first, 4 of 4)

- [A friendly introduction to triangular transport](https://mlsanity.com/paper/2503.21673/) (2025)
- [Optimal Experimental Design for Reliable Learning of History-Dependent Constitutive Laws](https://mlsanity.com/paper/2603.12365/) (2026)
- [Beyond Expected Information Gain: Stable Bayesian Optimal Experimental Design with Integral Probability Metrics and Plug-and-Play Extensions](https://mlsanity.com/paper/2604.21849/) (2026)
- [Multifidelity Formulations for Triangular Transport](https://mlsanity.com/paper/2610.00698/) (2026)

## Abstract (arXiv)

Computing expected information gain (EIG) from prior to posterior (equivalently, mutual information between candidate observations and model parameters or other quantities of interest) is a fundamental challenge in Bayesian optimal experimental design. We formulate flexible transport-based schemes for EIG estimation in general nonlinear/non-Gaussian settings, compatible with both standard and implicit Bayesian models. These schemes are representative of two-stage methods for estimating or bounding EIG using marginal and conditional density estimates. In this setting, we analyze the optimal allocation of samples between training (density estimation) and approximation of the outer prior expectation. We show that with this optimal sample allocation, the mean squared error (MSE) of the resulting EIG estimator converges more quickly than that of a standard nested Monte Carlo scheme. We then address the estimation of EIG in high dimensions, by deriving gradient-based upper bounds on the mutual information lost by projecting the parameters and/or observations to lower-dimensional subspaces. Minimizing these upper bounds yields projectors and hence low-dimensional EIG approximations that outperform approximations obtained via other linear dimension reduction schemes. Numerical experiments on a PDE-constrained Bayesian inverse problem also illustrate a favorable trade-off between dimension truncation and the modeling of non-Gaussianity, when estimating EIG from finite samples in high dimensions.