mlsanityData to 14 Oct 2026

Performance Comparisons of Reinforcement Learning Algorithms for Sequential Experimental Design

arXiv 2503.05905 ↗First posted 7 Mar 2025cs.LG stat.MLcited by 3

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Recent developments in sequential experimental design look to construct a policy that can efficiently navigate the design space, in a way that maximises the expected information gain. Whilst there is work on achieving tractable policies for experimental design problems, there is significantly less work on obtaining policies that are able to generalise well - i.e. able to give good performance despite a change in the underlying statistical properties of the experiments. Conducting experiments sequentially has recently brought about the use of reinforcement learning, where an agent is trained to navigate the design space to select the most informative designs for experimentation. However, there is still a lack of understanding about the benefits and drawbacks of using certain reinforcement learning algorithms to train these agents. In our work, we investigate several reinforcement learning algorithms and their efficacy in producing agents that take maximally informative design decisions in sequential experimental design scenarios. We find that agent performance is impacted depending on the algorithm used for training, and that particular algorithms, using dropout or ensemble approaches, empirically showcase attractive generalisation properties.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 28 of its 44 references that are in the map. First posted on arXiv 7 Mar 2025.

Builds on

28 of 44 references.

References · 28

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  5. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  6. 2018Pyro: Deep Universal Probabilistic Programmingcited by 380
  7. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  8. 2018An Introduction to Deep Reinforcement Learningcited by 206
  9. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  10. 2021A Fine-Grained Analysis on Distribution Shiftcited by 171
  11. 2019Variational Bayesian Optimal Experimental Designcited by 116
  12. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 114
16 more
  1. 2021Deep Adaptive Design: Amortizing Sequential Bayesian Experimental Designcited by 106
  2. 2017On Nesting Monte Carlo Estimatorscited by 83
  3. 2013Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizationscited by 70
  4. 2020Distributionally Robust Bayesian Optimizationcited by 61
  5. 2021Implicit Deep Adaptive Design: Policy-Based Experimental Design without Likelihoodscited by 61
  6. 2019A Unified Stochastic Gradient Approach to Designing Bayesian-Optimal Experimentscited by 58
  7. 2024Optimal experimental design: Formulations and computationscited by 45
  8. 2016Sequential Bayesian optimal experimental design via approximate dynamic programmingcited by 28
  9. 2021Gradient-based Bayesian Experimental Design for Implicit Models using Mutual Information Lower Boundscited by 27
  10. 2021Bayesian Sequential Optimal Experimental Design for Nonlinear Models Using Policy Gradient Reinforcement Learningcited by 24
  11. 2022Policy-Based Bayesian Experimental Design for Non-Differentiable Implicit Modelscited by 16
  12. 2022Robust Expected Information Gain for Optimal Bayesian Experimental Design Using Ambiguity Setscited by 12
  13. 2019Bayesian decision-theoretic design of experiments under an alternative modelcited by 8
  14. 2022Characterizing the robustness of Bayesian adaptive experimental designs to active learning biascited by 8
  15. 2023Metrics for Bayesian Optimal Experiment Design under Model Misspecificationcited by 7
  16. 2023Statistically Efficient Bayesian Sequential Experiment Design via Reinforcement Learning with Cross-Entropy Estimatorscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025ALINE: Joint Amortization for Bayesian Inference and Active Data Acquisitioncited by 8
  2. 2026Designing Time Series Experiments in A/B Testing with Transformer Reinforcement Learningcited by 1
  3. 2026FairBED: A Bayesian Experimental Design Approach to Gathering Fairer Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Position: Benchmarking is Limited in Reinforcement Learning Research
  2. 2025How Ensembles of Distilled Policies Improve Generalisation in Reinforcement Learning
  3. 2026Position: RL Researchers Need to Distinguish Between Solving Simulators and Using Simulators as a Proxy
  4. 2026Understanding Goal Generalisation in Sequential Reinforcement Learning
  5. 2024SPO: Sequential Monte Carlo Policy Optimisation
  6. 2026More Methods, Better Agents? A Reality Check on Self-Distillation in Agentic Reinforcement Learning
  7. 2026Deconstructing Actor-Critic: A Large-scale Empirical Study of Design Components for Practitioners
  8. 2024Performative Reinforcement Learning in Gradually Shifting Environments