mlsanityData to 7 Oct 2026

Safe Exploration via Policy Priors

arXiv 2601.19612 ↗First posted 27 Jan 2026cs.LG cs.AI cs.ROcited by 6

Not on the map. Why?

Abstract (arXiv)

Safe exploration is a key requirement for reinforcement learning (RL) agents to learn and adapt online, beyond controlled (e.g. simulated) environments. In this work, we tackle this challenge by utilizing suboptimal yet conservative policies (e.g., obtained from offline data or simulators) as priors. Our approach, SOOPER, uses probabilistic dynamics models to optimistically explore, yet pessimistically fall back to the conservative policy prior if needed. We prove that SOOPER guarantees safety throughout learning, and establish convergence to an optimal policy by bounding its cumulative regret. Extensive experiments on key safe RL benchmarks and real-world hardware demonstrate that SOOPER is scalable, outperforms the state-of-the-art and validate our theoretical guarantees in practice.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 68 of its 88 references that are in the map. First posted on arXiv 27 Jan 2026.

Builds on

68 of 88 references.

References · 68

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  4. 2018DeepMind Control Suitecited by 935
  5. 2017Constrained Policy Optimizationcited by 928
  6. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  7. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  8. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  9. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  10. 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 498
  11. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 417
  12. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
38 more
  1. 2017On Kernelized Multi-armed Banditscited by 348
  2. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
  3. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 337
  4. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278
  5. 2018Safe Exploration in Continuous Action Spacescited by 237
  6. 2020Recovery RL: Safe Reinforcement Learning with Learned Recovery Zonescited by 156
  7. 2016Bayesian Optimization with Safety Constraints: Safe and Automatic Parameter Tuning in Roboticscited by 134
  8. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 131
  9. 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 128
  10. 2019IPO: Interior-point Policy Optimization under Constraintscited by 119
  11. 2018Learning-based Model Predictive Control for Safe Explorationcited by 116
  12. 2020Safe Reinforcement Learning in Constrained Markov Decision Processescited by 109
  13. 2020Learning to be Safe: Deep RL with a Safety Criticcited by 109
  14. 2020Conservative Safety Critics for Explorationcited by 107
  15. 2024Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systemscited by 94
  16. 2017Cautious Model Predictive Control using Gaussian Process Regressioncited by 91
  17. 2018A predictive safety filter for learning-based control of constrained nonlinear dynamical systemscited by 91
  18. 2020Information Theoretic Regret Bounds for Online Nonlinear Controlcited by 87
  19. 2017Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learningcited by 86
  20. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexitycited by 85
  21. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexitycited by 84
  22. 2020Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planningcited by 83
  23. 2020An empirical investigation of the challenges of real-world reinforcement learningcited by 75
  24. 2019Adaptive and Safe Bayesian Optimization in High Dimensions via One-Dimensional Subspacescited by 73
  25. 2016Safe Exploration in Finite Markov Decision Processes with Gaussian Processescited by 72
  26. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 69
  27. 2022Safe Reinforcement Learning by Imagining the Near Futurecited by 67
  28. 2022Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentationcited by 63
  29. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 62
  30. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 58
  31. 2022Constrained Policy Optimization via Bayesian World Modelscited by 55
  32. 2023SafeDreamer: Safe Reinforcement Learning with World Modelscited by 50
  33. 2021Safe Reinforcement Learning Using Advantage-Based Interventioncited by 45
  34. 2021DOPE: Doubly Optimistic and Pessimistic Exploration for Safe Reinforcement Learningcited by 34
  35. 2023Optimistic Active Exploration of Dynamical Systemscited by 32
  36. 2019AMZ Driverless: The Full Autonomous Racing Systemcited by 28
  37. 2023Safe Exploration in Reinforcement Learning: A Generalized Formulation and Algorithmscited by 27
  38. 2024ActSafe: Active Exploration with Safety Constraints for Reinforcement Learningcited by 22

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Golden Handcuffs make safer AI agentscited by 1
  2. 2026Global Optimality for Constrained Exploration via Penalty Regularizationcited by 0
  3. 2026Sampling-Based Safe Reinforcement Learningcited by 0
  4. 2026Directional Constraints for Efficient Exploration in Safe Reinforcement Learningcited by 0
  5. 2026Provably Safe Sim-to-Real Transfercited by 0
  6. 2026From Static Policies to Adaptive Priors in Offline Reinforcement Learningcited by 0