# Variational Sequential Optimal Experimental Design using Reinforcement Learning

arXiv: https://arxiv.org/abs/2306.10430 · HTML: https://mlsanity.com/paper/2306.10430/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 89 references that are in the map.
- First posted on arXiv 17 Jun 2023.

## Builds on (most cited first, 30 of 35)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Randomized Ensembled Double Q-Learning: Learning Fast Without a Model](https://mlsanity.com/paper/2101.05982/) (2021)
- [Analyzing Inverse Problems with Invertible Neural Networks](https://mlsanity.com/paper/1808.04730/) (2018)
- [BayesFlow: Learning complex stochastic models with invertible neural networks](https://mlsanity.com/paper/2003.06281/) (2020)
- [Modern Bayesian Experimental Design](https://mlsanity.com/paper/2302.14545/) (2023)
- [Variational Bayesian Optimal Experimental Design](https://mlsanity.com/paper/1903.05480/) (2019)
- [Deep Adaptive Design: Amortizing Sequential Bayesian Experimental Design](https://mlsanity.com/paper/2103.02438/) (2021)
- [Amortized Inference for Causal Structure Learning](https://mlsanity.com/paper/2205.12934/) (2022)
- [Prediction-Oriented Bayesian Active Learning](https://mlsanity.com/paper/2304.08151/) (2023)
- [Implicit Deep Adaptive Design: Policy-Based Experimental Design without Likelihoods](https://mlsanity.com/paper/2111.02329/) (2021)
- [A Unified Stochastic Gradient Approach to Designing Bayesian-Optimal Experiments](https://mlsanity.com/paper/1911.00294/) (2019)
- [Bayesian Experimental Design for Implicit Models by Mutual Information Neural Estimation](https://mlsanity.com/paper/2002.08129/) (2020)
- [Optimizing Sequential Experimental Design with Deep Reinforcement Learning](https://mlsanity.com/paper/2202.00821/) (2022)
- [Solving inverse problems using conditional invertible neural networks](https://mlsanity.com/paper/2007.15849/) (2020)
- [HINT: Hierarchical Invertible Neural Transport for Density Estimation and Bayesian Inference](https://mlsanity.com/paper/1905.10687/) (2019)
- [Sequential Bayesian optimal experimental design via approximate dynamic programming](https://mlsanity.com/paper/1604.08320/) (2016)
- [Gradient-based Bayesian Experimental Design for Implicit Models using Mutual Information Lower Bounds](https://mlsanity.com/paper/2105.04379/) (2021)
- [Bayesian Sequential Optimal Experimental Design for Nonlinear Models Using Policy Gradient Reinforcement Learning](https://mlsanity.com/paper/2110.15335/) (2021)
- [Sequential Bayesian Experimental Design for Implicit Models via Mutual Information](https://mlsanity.com/paper/2003.09379/) (2020)

## Built on by (most cited first, 8 of 8)

- [ALINE: Joint Amortization for Bayesian Inference and Active Data Acquisition](https://mlsanity.com/paper/2506.07259/) (2025)
- [JADAI: Jointly Amortizing Adaptive Design and Bayesian Inference](https://mlsanity.com/paper/2512.22999/) (2025)
- [A Likelihood-Free Approach to Goal-Oriented Bayesian Optimal Experimental Design](https://mlsanity.com/paper/2408.09582/) (2024)
- [Optimizing Likelihoods via Mutual Information: Bridging Simulation-Based Inference and Bayesian Optimal Experimental Design](https://mlsanity.com/paper/2502.08004/) (2025)
- [Goal-Oriented Sequential Bayesian Experimental Design for Causal Learning](https://mlsanity.com/paper/2507.07359/) (2025)
- [Constrained Bayesian Experimental Design via Online Planning](https://mlsanity.com/paper/2605.26990/) (2026)
- [Efficient Adaptive Data Acquisition via Pretrained Belief Representations](https://mlsanity.com/paper/2606.25197/) (2026)
- [Sequential Bayesian Optimal Experimental Design in Infinite Dimensions via Policy Gradient Reinforcement Learning](https://mlsanity.com/paper/2601.05868/) (2026)

## Abstract (arXiv)

We present variational sequential optimal experimental design (vsOED), a novel method for optimally designing a finite sequence of experiments within a Bayesian framework with information-theoretic criteria. vsOED employs a one-point reward formulation with variational posterior approximations, providing a provable lower bound to the expected information gain. Numerical methods are developed following an actor-critic reinforcement learning approach, including derivation and estimation of variational and policy gradients to optimize the design policy, and posterior approximation using Gaussian mixture models and normalizing flows. vsOED accommodates nuisance parameters, implicit likelihoods, and multiple candidate models, while supporting flexible design criteria that can target designs for model discrimination, parameter inference, goal-oriented prediction, and their weighted combinations. We demonstrate vsOED across various engineering and science applications, illustrating its superior sample efficiency compared to existing sequential experimental design algorithms.