mlsanityData to 14 Oct 2026

Uncertainty-Aware Selection of Online Algorithms with Simulator Ensembles

arXiv 2609.32170 ↗First posted 26 Sep 2026cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

The performance of online reinforcement learning depends critically on design choices, especially those that affect exploration. These choices are often selected by fitting a simulator to offline data, evaluating candidate algorithms in that simulator, and deploying the best-performing one. The simplest Plug-In selection rule simply selects the best performing algorithm on the fitted simulator, making evaluations unreliable when the offline data used to fit the simulator are limited. We investigate Uncertainty-Aware selection, which forms an ensemble of simulators---for example, obtained by bootstrap resampling---and selects the online algorithm with the best average performance across the ensemble. While ensemble-based approaches have been used to mitigate distribution shift and facilitate sim-to-real transfer, we formally show that this approach can mitigate the effects of limited data when fitting the simulator and theoretically has significant regret gains compared to Plug-In selection in multi-armed bandits. We also empirically investigate the Uncertainty-Aware selection approach in deep RL experiments on robotic control tasks that involve selecting reward-shaping hyperparameters, and show that it leads to more reliable selection and improved online performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 10 of its 74 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

10 of 74 references.

References · 10

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2019Challenges of Real-World Reinforcement Learningcited by 291
  3. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 164
  4. 2019Distributionally Robust Reinforcement Learningcited by 42
  5. 2025Uncertainty-Aware Robotic World Model Makes Offline Model-Based Reinforcement Learning Work on Real Robotscited by 17
  6. 2022The Typical Behavior of Bandit Algorithmscited by 12
  7. 2022Towards Data-Driven Offline Simulations for Online Reinforcement Learningcited by 3
  8. 2024Oralytics Reinforcement Learning Algorithmcited by 2
  9. 2025Automatic Reward Shaping from Multi-Objective Human Heuristicscited by 1
  10. 2026A Diffusion-Model Subpopulation Digital Twin for Mobile Health Deployment: A Case Study on the HeartSteps Interventioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learning
  2. 2025Improved Training Mechanism for Reinforcement Learning via Online Model Selection
  3. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
  4. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness
  5. 2026Epistemic Robust Offline Reinforcement Learning
  6. 2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning
  7. 2026Theoretical Foundations and Effective Algorithms for Policy-Aware Simulator Learning
  8. 2024Batch Ensemble for Variance Dependent Regret in Stochastic Bandits