mlsanityData to 14 Oct 2026

SHAPO: Sharpness-Aware Policy Optimization for Safe Exploration

arXiv 2606.10228 ↗First posted 8 Jun 2026cs.LG cs.AI cs.ROcited by 2

On the map: Deep reinforcement learning methods → Safe and constrained RL

Abstract (arXiv)

Safe exploration is a prerequisite for deploying reinforcement learning (RL) agents in safety-critical domains. In this paper, we approach safe exploration through the lens of epistemic uncertainty, where the actor's sensitivity to parameter perturbations serves as a practical proxy for regions of high uncertainty. We propose Sharpness-Aware Policy Optimization (SHAPO), a sharpness-aware policy update rule that evaluates gradients at perturbed parameters, making policy updates pessimistic with respect to the actor's epistemic uncertainty. Analytically we show that this adjustment implicitly reweighs policy gradients, amplifying the influence of rare unsafe actions while tempering contributions from already safe ones, thereby biasing learning toward conservative behavior in under-explored regions. Across several continuous-control tasks, our method consistently improves both safety and task performance over existing baselines, significantly expanding their Pareto frontiers.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 32 of its 46 references that are in the map. First posted on arXiv 8 Jun 2026.

Builds on

32 of 46 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Safety Representations for Safer Policy Learning27 Feb

Other references · 31

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  5. 2015Trust Region Policy Optimizationcited by 2,580
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  7. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  8. 2017Constrained Policy Optimizationcited by 983
  9. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  10. 2016Deep Exploration via Bootstrapped DQNcited by 527
  11. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  12. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
19 more
  1. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  2. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 219
  3. 2022Towards Understanding Sharpness-Aware Minimizationcited by 146
  4. 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 134
  5. 2020Learning to be Safe: Deep RL with a Safety Criticcited by 116
  6. 2020Conservative Safety Critics for Explorationcited by 113
  7. 2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
  8. 2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 80
  9. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 71
  10. 2022Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentationcited by 69
  11. 2023SafeDreamer: Safe Reinforcement Learning with World Modelscited by 54
  12. 2022SAM as an Optimal Relaxation of Bayescited by 41
  13. 2022Lyapunov Density Models: Constraining Distribution Shift in Learning-Based Controlcited by 28
  14. 2022Sample Efficient Deep Reinforcement Learning via Uncertainty Estimationcited by 27
  15. 2024Why is SAM Robust to Label Noise?cited by 21
  16. 2023Policy Optimization in a Noisy Neighborhood: On Return Landscapes in Continuous Controlcited by 11
  17. 2023Value-Distributional Model-Based Reinforcement Learningcited by 5
  18. 2023Solving Richly Constrained Reinforcement Learning through State Augmentation and Reward Penaltiescited by 4
  19. 2022How to Enable Uncertainty Estimation in Proximal Policy Optimizationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Learning Safe Agent Behaviour from Human Preferences and Justifications via World Modelscited by 0
  2. 2026Learning Perturbation Robust Policies for LLM Agents with Stable Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Safe But Not Sorry: Reducing Over-Conservatism in Safety Critics via Uncertainty-Aware Modulation
  2. 2026Safe Exploration via Policy Priors
  3. 2026Sampling-Based Safe Reinforcement Learning
  4. 2026Deep Epistemic Value Functions for Optimistic Exploration
  5. 2026Bayesian Conservative Policy Optimization (BCPO): A Novel Uncertainty-Calibrated Offline Reinforcement Learning with Credible Lower Bounds
  6. 2025Overcoming Non-stationary Dynamics with Evidential Proximal Policy Optimization
  7. 2024Safe Exploration Using Bayesian World Models and Log-Barrier Optimization
  8. 2026CAPSULE: Control-Theoretic Action Perturbations for Safe Uncertainty-Aware Reinforcement Learning