mlsanityData to 14 Oct 2026

Bayesian Sequential Optimal Experimental Design for Nonlinear Models Using Policy Gradient Reinforcement Learning

arXiv 2110.15335 ↗First posted 28 Oct 2021cs.LG stat.CO stat.MEcited by 24

Not on the map. Why?

Abstract (arXiv)

We present a mathematical framework and computational methods to optimally design a finite number of sequential experiments. We formulate this sequential optimal experimental design (sOED) problem as a finite-horizon partially observable Markov decision process (POMDP) in a Bayesian setting and with information-theoretic utilities. It is built to accommodate continuous random variables, general non-Gaussian posteriors, and expensive nonlinear forward models. sOED then seeks an optimal design policy that incorporates elements of both feedback and lookahead, generalizing the suboptimal batch and greedy designs. We solve for the sOED policy numerically via policy gradient (PG) methods from reinforcement learning, and derive and prove the PG expression for sOED. Adopting an actor-critic approach, we parameterize the policy and value functions using deep neural networks and improve them using gradient estimates produced from simulated episodes of designs and observations. The overall PG-sOED method is validated on a linear-Gaussian benchmark, and its advantages over batch and greedy designs are demonstrated through a contaminant source inversion problem in a convection-diffusion field.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 24 of its 86 references that are in the map. First posted on arXiv 28 Oct 2021.

Builds on

24 of 86 references.

References · 24

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  5. 2015Trust Region Policy Optimizationcited by 2,580
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  7. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  8. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  9. 2016Variational Inference: A Review for Statisticianscited by 1,472
  10. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  11. 2017Deep Reinforcement Learning: An Overviewcited by 333
  12. 2017Noisy Networks for Explorationcited by 266
12 more
  1. 2018Distributed Distributional Deterministic Policy Gradientscited by 199
  2. 2017Parameter Space Noise for Explorationcited by 161
  3. 2018Deep Variational Reinforcement Learning for POMDPscited by 125
  4. 2019Variational Bayesian Optimal Experimental Designcited by 116
  5. 2016Likelihood-free inference by ratio estimationcited by 81
  6. 2017Stein Variational Policy Gradientcited by 44
  7. 2018Efficient Bayesian Experimental Design for Implicit Modelscited by 34
  8. 2016Sequential Bayesian optimal experimental design via approximate dynamic programmingcited by 28
  9. 2020Sequential Bayesian Experimental Design for Implicit Models via Mutual Informationcited by 23
  10. 2014A Fast and Scalable Method for A-Optimal Design of Experiments for Infinite-dimensional Bayesian Nonlinear Inverse Problemscited by 14
  11. 2020A fast and scalable computational framework for large-scale and high-dimensional Bayesian optimal experimental designcited by 9
  12. 2015Efficient Bayesian experimentation using an expected information gain lower boundcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 4; 2023: 3; 2024: 4; 2025: 6; 2026: 6.

24 papers.

Sort:

By area:

  1. 2023Modern Bayesian Experimental Designcited by 125
  2. 2022Optimizing Sequential Experimental Design with Deep Reinforcement Learningcited by 46
  3. 2022A Comprehensive Review of Digital Twin -- Part 2: Roles of Uncertainty Quantification and Optimization, a Battery Digital Twin, and Perspectivescited by 18
  4. 2022Policy-Based Bayesian Experimental Design for Non-Differentiable Implicit Modelscited by 16
  5. 2024Variational Bayesian Optimal Experimental Design with Normalizing Flowscited by 13
  6. 2022Illusory Attacks: Information-Theoretic Detectability Matters in Adversarial Attackscited by 12
  7. 2025Step-DAD: Semi-Amortized Policy-Based Bayesian Experimental Designcited by 12
  8. 2023Variational Sequential Optimal Experimental Design using Reinforcement Learningcited by 8
  9. 2025Active Learning of Model Discrepancy with Bayesian Experimental Designcited by 5
  10. 2023Experimentation Platforms Meet Reinforcement Learning: Bayesian Sequential Decision-Making for Continuous Monitoringcited by 3
  11. 2025Performance Comparisons of Reinforcement Learning Algorithms for Sequential Experimental Designcited by 3
  12. 2024Amortized nonmyopic active search via deep imitation learningcited by 2
12 more
  1. 2025Bayesian Experimental Design for Model Discrepancy Calibration: An Auto-Differentiable Ensemble Kalman Inversion Approachcited by 2
  2. 2025Goal-Oriented Sequential Bayesian Experimental Design for Causal Learningcited by 2
  3. 2026Action-BED: Task-Driven Bayesian Experimental Design with Singly Intractable Objectivescited by 2
  4. 2021Reinforcement Learning based Sequential Batch-sampling for Bayesian Optimal Experimental Designcited by 1
  5. 2024Asynchronous Parallel Reinforcement Learning for Optimizing Propulsive Performance in Fin Ray Controlcited by 1
  6. 2024Uncertainty Quantification of Graph Convolution Neural Network Models of Evolving Processescited by 1
  7. 2025Dynamic Angle Selection in X-Ray CT: A Reinforcement Learning Approach to Optimal Stoppingcited by 1
  8. 2026Sequential Bayesian Optimal Experimental Design in Infinite Dimensions via Policy Gradient Reinforcement Learningcited by 1
  9. 2026Bayesian Experimental Design via Score Matchingcited by 1
  10. 2026Bayesian Experimental Design for Model Discrepancy Calibration: A Rivalry between Kullback--Leibler Divergence and Wasserstein Distancecited by 0
  11. 2026Bayesian experimental design: grouped geometric pooled posterior via ensemble Kalman methodscited by 0
  12. 2026Information-theoretic receding-horizon active learning of nonlinear dynamical systemscited by 0