# Statistically Efficient Bayesian Sequential Experiment Design via Reinforcement Learning with Cross-Entropy Estimators

arXiv: https://arxiv.org/abs/2305.18435 · HTML: https://mlsanity.com/paper/2305.18435/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 50 references that are in the map.
- First posted on arXiv 29 May 2023.

## Builds on (most cited first, 27 of 27)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [The frontier of simulation-based inference](https://mlsanity.com/paper/1911.01429/) (2019)
- [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018)
- [Randomized Ensembled Double Q-Learning: Learning Fast Without a Model](https://mlsanity.com/paper/2101.05982/) (2021)
- [Formal Limitations on the Measurement of Mutual Information](https://mlsanity.com/paper/1811.04251/) (2018)
- [Deep Reinforcement Learning and the Deadly Triad](https://mlsanity.com/paper/1812.02648/) (2018)
- [Learning Likelihoods with Conditional Normalizing Flows](https://mlsanity.com/paper/1912.00042/) (2019)
- [Modern Bayesian Experimental Design](https://mlsanity.com/paper/2302.14545/) (2023)
- [Variational Bayesian Optimal Experimental Design](https://mlsanity.com/paper/1903.05480/) (2019)
- [Conditioning by adaptive sampling for robust design](https://mlsanity.com/paper/1901.10060/) (2019)
- [Deep Adaptive Design: Amortizing Sequential Bayesian Experimental Design](https://mlsanity.com/paper/2103.02438/) (2021)
- [Model-Based Active Exploration](https://mlsanity.com/paper/1810.12162/) (2018)
- [On Nesting Monte Carlo Estimators](https://mlsanity.com/paper/1709.06181/) (2017)
- [Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizations](https://mlsanity.com/paper/1308.3513/) (2013)
- [Implicit Deep Adaptive Design: Policy-Based Experimental Design without Likelihoods](https://mlsanity.com/paper/2111.02329/) (2021)
- [A Unified Stochastic Gradient Approach to Designing Bayesian-Optimal Experiments](https://mlsanity.com/paper/1911.00294/) (2019)
- [Bayesian Experimental Design for Implicit Models by Mutual Information Neural Estimation](https://mlsanity.com/paper/2002.08129/) (2020)
- [normflows: A PyTorch Package for Normalizing Flows](https://mlsanity.com/paper/2302.12014/) (2023)
- [ABCD-Strategy: Budgeted Experimental Design for Targeted Causal Structure Discovery](https://mlsanity.com/paper/1902.10347/) (2019)
- [Optimizing Sequential Experimental Design with Deep Reinforcement Learning](https://mlsanity.com/paper/2202.00821/) (2022)
- [BINOCULARS for Efficient, Nonmyopic Sequential Experimental Design](https://mlsanity.com/paper/1909.04568/) (2019)
- [Why Non-myopic Bayesian Optimization is Promising and How Far Should We Look-ahead? A Study via Rollout](https://mlsanity.com/paper/1911.01004/) (2019)
- [Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE](https://mlsanity.com/paper/2107.01152/) (2021)
- [Flow-based Recurrent Belief State Learning for POMDPs](https://mlsanity.com/paper/2205.11051/) (2022)

## Built on by (most cited first, 6 of 6)

- [JADAI: Jointly Amortizing Adaptive Design and Bayesian Inference](https://mlsanity.com/paper/2512.22999/) (2025)
- [Performance Comparisons of Reinforcement Learning Algorithms for Sequential Experimental Design](https://mlsanity.com/paper/2503.05905/) (2025)
- [Optimizing Likelihoods via Mutual Information: Bridging Simulation-Based Inference and Bayesian Optimal Experimental Design](https://mlsanity.com/paper/2502.08004/) (2025)
- [Constrained Bayesian Experimental Design via Online Planning](https://mlsanity.com/paper/2605.26990/) (2026)
- [Efficient Adaptive Data Acquisition via Pretrained Belief Representations](https://mlsanity.com/paper/2606.25197/) (2026)
- [Deep adaptive design with an evidential bias criterion](https://mlsanity.com/paper/2608.16466/) (2026)

## Abstract (arXiv)

Reinforcement learning can learn amortised design policies for designing sequences of experiments. However, current amortised methods rely on estimators of expected information gain (EIG) that require an exponential number of samples on the magnitude of the EIG to achieve an unbiased estimation. We propose the use of an alternative estimator based on the cross-entropy of the joint model distribution and a flexible proposal distribution. This proposal distribution approximates the true posterior of the model parameters given the experimental history and the design policy. Our method overcomes the exponential-sample complexity of previous approaches and provide more accurate estimates of high EIG values. More importantly, it allows learning of superior design policies, and is compatible with continuous and discrete design spaces, non-differentiable likelihoods and even implicit probabilistic models.