# Bayesian Sequential Optimal Experimental Design for Nonlinear Models Using Policy Gradient Reinforcement Learning

arXiv: https://arxiv.org/abs/2110.15335 · HTML: https://mlsanity.com/paper/2110.15335/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 86 references that are in the map.
- First posted on arXiv 28 Oct 2021.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Deep Reinforcement Learning: An Overview](https://mlsanity.com/paper/1701.07274/) (2017)
- [Noisy Networks for Exploration](https://mlsanity.com/paper/1706.10295/) (2017)
- [Distributed Distributional Deterministic Policy Gradients](https://mlsanity.com/paper/1804.08617/) (2018)
- [Parameter Space Noise for Exploration](https://mlsanity.com/paper/1706.01905/) (2017)
- [Deep Variational Reinforcement Learning for POMDPs](https://mlsanity.com/paper/1806.02426/) (2018)
- [Variational Bayesian Optimal Experimental Design](https://mlsanity.com/paper/1903.05480/) (2019)
- [Likelihood-free inference by ratio estimation](https://mlsanity.com/paper/1611.10242/) (2016)
- [Stein Variational Policy Gradient](https://mlsanity.com/paper/1704.02399/) (2017)
- [Efficient Bayesian Experimental Design for Implicit Models](https://mlsanity.com/paper/1810.09912/) (2018)
- [Sequential Bayesian optimal experimental design via approximate dynamic programming](https://mlsanity.com/paper/1604.08320/) (2016)
- [Sequential Bayesian Experimental Design for Implicit Models via Mutual Information](https://mlsanity.com/paper/2003.09379/) (2020)
- [A Fast and Scalable Method for A-Optimal Design of Experiments for Infinite-dimensional Bayesian Nonlinear Inverse Problems](https://mlsanity.com/paper/1410.5899/) (2014)
- [A fast and scalable computational framework for large-scale and high-dimensional Bayesian optimal experimental design](https://mlsanity.com/paper/2010.15196/) (2020)
- [Efficient Bayesian experimentation using an expected information gain lower bound](https://mlsanity.com/paper/1506.00053/) (2015)

## Built on by (most cited first, 24 of 24)

- [Modern Bayesian Experimental Design](https://mlsanity.com/paper/2302.14545/) (2023)
- [Optimizing Sequential Experimental Design with Deep Reinforcement Learning](https://mlsanity.com/paper/2202.00821/) (2022)
- [A Comprehensive Review of Digital Twin -- Part 2: Roles of Uncertainty Quantification and Optimization, a Battery Digital Twin, and Perspectives](https://mlsanity.com/paper/2208.12904/) (2022)
- [Policy-Based Bayesian Experimental Design for Non-Differentiable Implicit Models](https://mlsanity.com/paper/2203.04272/) (2022)
- [Variational Bayesian Optimal Experimental Design with Normalizing Flows](https://mlsanity.com/paper/2404.13056/) (2024)
- [Illusory Attacks: Information-Theoretic Detectability Matters in Adversarial Attacks](https://mlsanity.com/paper/2207.10170/) (2022)
- [Step-DAD: Semi-Amortized Policy-Based Bayesian Experimental Design](https://mlsanity.com/paper/2507.14057/) (2025)
- [Variational Sequential Optimal Experimental Design using Reinforcement Learning](https://mlsanity.com/paper/2306.10430/) (2023)
- [Active Learning of Model Discrepancy with Bayesian Experimental Design](https://mlsanity.com/paper/2502.05372/) (2025)
- [Experimentation Platforms Meet Reinforcement Learning: Bayesian Sequential Decision-Making for Continuous Monitoring](https://mlsanity.com/paper/2304.00420/) (2023)
- [Performance Comparisons of Reinforcement Learning Algorithms for Sequential Experimental Design](https://mlsanity.com/paper/2503.05905/) (2025)
- [Amortized nonmyopic active search via deep imitation learning](https://mlsanity.com/paper/2405.15031/) (2024)
- [Bayesian Experimental Design for Model Discrepancy Calibration: An Auto-Differentiable Ensemble Kalman Inversion Approach](https://mlsanity.com/paper/2504.20319/) (2025)
- [Goal-Oriented Sequential Bayesian Experimental Design for Causal Learning](https://mlsanity.com/paper/2507.07359/) (2025)
- [Action-BED: Task-Driven Bayesian Experimental Design with Singly Intractable Objectives](https://mlsanity.com/paper/2606.23662/) (2026)
- [Reinforcement Learning based Sequential Batch-sampling for Bayesian Optimal Experimental Design](https://mlsanity.com/paper/2112.10944/) (2021)
- [Asynchronous Parallel Reinforcement Learning for Optimizing Propulsive Performance in Fin Ray Control](https://mlsanity.com/paper/2401.11349/) (2024)
- [Uncertainty Quantification of Graph Convolution Neural Network Models of Evolving Processes](https://mlsanity.com/paper/2402.11179/) (2024)
- [Dynamic Angle Selection in X-Ray CT: A Reinforcement Learning Approach to Optimal Stopping](https://mlsanity.com/paper/2503.12688/) (2025)
- [Sequential Bayesian Optimal Experimental Design in Infinite Dimensions via Policy Gradient Reinforcement Learning](https://mlsanity.com/paper/2601.05868/) (2026)
- [Bayesian Experimental Design via Score Matching](https://mlsanity.com/paper/2607.08335/) (2026)
- [Bayesian Experimental Design for Model Discrepancy Calibration: A Rivalry between Kullback--Leibler Divergence and Wasserstein Distance](https://mlsanity.com/paper/2601.16425/) (2026)
- [Bayesian experimental design: grouped geometric pooled posterior via ensemble Kalman methods](https://mlsanity.com/paper/2604.18505/) (2026)
- [Information-theoretic receding-horizon active learning of nonlinear dynamical systems](https://mlsanity.com/paper/2609.36712/) (2026)

## Abstract (arXiv)

We present a mathematical framework and computational methods to optimally design a finite number of sequential experiments. We formulate this sequential optimal experimental design (sOED) problem as a finite-horizon partially observable Markov decision process (POMDP) in a Bayesian setting and with information-theoretic utilities. It is built to accommodate continuous random variables, general non-Gaussian posteriors, and expensive nonlinear forward models. sOED then seeks an optimal design policy that incorporates elements of both feedback and lookahead, generalizing the suboptimal batch and greedy designs. We solve for the sOED policy numerically via policy gradient (PG) methods from reinforcement learning, and derive and prove the PG expression for sOED. Adopting an actor-critic approach, we parameterize the policy and value functions using deep neural networks and improve them using gradient estimates produced from simulated episodes of designs and observations. The overall PG-sOED method is validated on a linear-Gaussian benchmark, and its advantages over batch and greedy designs are demonstrated through a contaminant source inversion problem in a convection-diffusion field.