mlsanityData to 7 Oct 2026

Optimizing Sequential Experimental Design with Deep Reinforcement Learning

arXiv 2202.00821 ↗First posted 2 Feb 2022cs.LG stat.MLcited by 45

Not on the map. Why?

Abstract (arXiv)

Bayesian approaches developed to solve the optimal design of sequential experiments are mathematically elegant but computationally challenging. Recently, techniques using amortization have been proposed to make these Bayesian approaches practical, by training a parameterized policy that proposes designs efficiently at deployment time. However, these methods may not sufficiently explore the design space, require access to a differentiable probabilistic model and can only optimize over continuous design spaces. Here, we address these limitations by showing that the problem of optimizing policies can be reduced to solving a Markov decision process (MDP). We solve the equivalent MDP with modern deep reinforcement learning techniques. Our experiments show that our approach is also computationally efficient at deployment time and exhibits state-of-the-art performance on both continuous and discrete design spaces, even when the probabilistic model is a black box.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 19 of its 52 references that are in the map. First posted on arXiv 2 Feb 2022.

Builds on

19 of 52 references.

References · 19

  1. 2017Deep Bayesian Active Learning with Image Datacited by 812
  2. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  3. 2018Pyro: Deep Universal Probabilistic Programmingcited by 379
  4. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  5. 2019Variational Bayesian Optimal Experimental Designcited by 116
  6. 2018Model-Based Active Explorationcited by 105
  7. 2021Deep Adaptive Design: Amortizing Sequential Bayesian Experimental Designcited by 103
  8. 2017On Nesting Monte Carlo Estimatorscited by 82
  9. 2013Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizationscited by 68
  10. 2019Shaping Belief States with Generative Environment Models for RLcited by 65
  11. 2021Implicit Deep Adaptive Design: Policy-Based Experimental Design without Likelihoodscited by 61
  12. 2019A Unified Stochastic Gradient Approach to Designing Bayesian-Optimal Experimentscited by 58
7 more
  1. 2020Bayesian Experimental Design for Implicit Models by Mutual Information Neural Estimationcited by 56
  2. 2021MADE: Exploration via Maximizing Deviation from Explored Regionscited by 40
  3. 2018Efficient Bayesian Experimental Design for Implicit Modelscited by 34
  4. 2016Sequential Bayesian optimal experimental design via approximate dynamic programmingcited by 27
  5. 2020Sequential Bayesian Experimental Design for Implicit Models via Mutual Informationcited by 23
  6. 2021Bayesian Sequential Optimal Experimental Design for Nonlinear Models Using Policy Gradient Reinforcement Learningcited by 22
  7. 2018Multilevel Monte Carlo estimation of expected information gainscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 9; 2024: 9; 2025: 13; 2026: 12.

45 papers.

Sort:

By area:

  1. 2023Modern Bayesian Experimental Designcited by 121
  2. 2023GFlowNets for AI-Driven Scientific Discoverycited by 63
  3. 2025BED-LLM: Intelligent Information Gathering with LLMs and Bayesian Experimental Designcited by 21
  4. 2024Amortized Bayesian Experimental Design for Decision-Makingcited by 20
  5. 2022Policy-Based Bayesian Experimental Design for Non-Differentiable Implicit Modelscited by 15
  6. 2023Differentiable Multi-Target Causal Bayesian Experimental Designcited by 15
  7. 2024Nesting Particle Filters for Experimental Design in Dynamical Systemscited by 12
  8. 2025Step-DAD: Semi-Amortized Policy-Based Bayesian Experimental Designcited by 12
  9. 2024Amortized Active Causal Induction with Deep Reinforcement Learningcited by 10
  10. 2024Bayesian Experimental Design via Contrastive Diffusionscited by 9
  11. 2023Variational Sequential Optimal Experimental Design using Reinforcement Learningcited by 8
  12. 2025ALINE: Joint Amortization for Bayesian Inference and Active Data Acquisitioncited by 8
33 more
  1. 2025PABBO: Preferential Amortized Black-Box Optimizationcited by 7
  2. 2024Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experimentscited by 6
  3. 2023Statistically Efficient Bayesian Sequential Experiment Design via Reinforcement Learning with Cross-Entropy Estimatorscited by 5
  4. 2023Bayesian Active Learning in the Presence of Nuisance Parameterscited by 5
  5. 2024PASOA- PArticle baSed Bayesian Optimal Adaptive designcited by 5
  6. 2025Balancing Interference and Correlation in Spatial Experimental Designs: A Causal Graph Cut Approachcited by 5
  7. 2023Online simulator-based experimental design for cognitive model selectioncited by 3
  8. 2024Amortized nonmyopic active search via deep imitation learningcited by 2
  9. 2024Adapting Projection-Based Reduced-Order Models using Projected Gaussian Processcited by 2
  10. 2025Optimizing Likelihoods via Mutual Information: Bridging Simulation-Based Inference and Bayesian Optimal Experimental Designcited by 2
  11. 2025A Geometric Approach to Optimal Experimental Designcited by 2
  12. 2025JADAI: Jointly Amortizing Adaptive Design and Bayesian Inferencecited by 2
  13. 2026Constrained Bayesian Experimental Design via Online Planningcited by 2
  14. 2026Efficient Adaptive Data Acquisition via Pretrained Belief Representationscited by 2
  15. 2023Amortised Experimental Design and Parameter Estimation for User Models of Pointingcited by 1
  16. 2024Bayesian Adaptive Calibration and Optimal Designcited by 1
  17. 2025Dynamic Angle Selection in X-Ray CT: A Reinforcement Learning Approach to Optimal Stoppingcited by 1
  18. 2025Goal-Oriented Sequential Bayesian Experimental Design for Causal Learningcited by 1
  19. 2025From Sorting Algorithms to Scalable Kernels: Bayesian Optimization in High-Dimensional Permutation Spacescited by 1
  20. 2026Sequential Bayesian Optimal Experimental Design in Infinite Dimensions via Policy Gradient Reinforcement Learningcited by 1
  21. 2026Designing Time Series Experiments in A/B Testing with Transformer Reinforcement Learningcited by 1
  22. 2026Supercharging Simulation-Based Inference for Bayesian Optimal Experimental Designcited by 1
  23. 2026When Should an AI Scientist Stop? Verifiable Experiment Steering and Refusal for Autonomous Discoverycited by 1
  24. 2022Experimental Design for Multi-Channel Imaging via Task-Driven Feature Selectioncited by 0
  25. 2023Amortised Design Optimization for Item Response Theorycited by 0
  26. 2025Active MRI Acquisition with Diffusion Guided Bayesian Experimental Designcited by 0
  27. 2025Direct Regret Optimization in Bayesian Optimizationcited by 0
  28. 2026Case-Guided Sequential Assay Planning in Drug Discoverycited by 0
  29. 2026Maximin Robust Bayesian Experimental Designcited by 0
  30. 2026Information-Theoretic Optimization for Task-Adapted Compressed Sensing Magnetic Resonance Imagingcited by 0
  31. 2026SymQNet: Amortized Acquisition for Low-Latency Adaptive Hamiltonian Learningcited by 0
  32. 2026FairBED: A Bayesian Experimental Design Approach to Gathering Fairer Datacited by 0
  33. 2026Bayesian Experimental Design via Score Matchingcited by 0