mlsanityData to 14 Oct 2026

Optimistic Exploration for Risk-Averse Constrained Reinforcement Learning

arXiv 2507.08793 ↗First posted 11 Jul 2025cs.LG cs.AIcited by 3

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Risk-averse Constrained Reinforcement Learning (RaCRL) aims to learn policies that minimise the likelihood of rare and catastrophic constraint violations caused by an environment's inherent randomness. In general, risk-aversion leads to conservative exploration of the environment which typically results in converging to sub-optimal policies that fail to adequately maximise reward or, in some cases, fail to achieve the goal. In this paper, we propose an exploration-based approach for RaCRL called Optimistic Risk-averse Actor Critic (ORAC), which constructs an exploratory policy by maximising a local upper confidence bound of the state-action reward value function whilst minimising a local lower confidence bound of the risk-averse state-action cost value function. Specifically, at each step, the weighting assigned to the cost value is increased or decreased if it exceeds or falls below the safety constraint value. This way the policy is encouraged to explore uncertain regions of the environment to discover high reward states whilst still satisfying the safety constraints. Our experimental results demonstrate that the ORAC approach prevents convergence to sub-optimal policies and improves significantly the reward-cost trade-off in various continuous control tasks such as Safety-Gymnasium and a complex building energy management environment CityLearn.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 25 of its 37 references that are in the map. First posted on arXiv 11 Jul 2025.

Builds on

25 of 37 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Spectral-Risk Safe Reinforcement Learning with Convergence Guarantees29 May

Other references · 24

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  3. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  4. 2017Constrained Policy Optimizationcited by 983
  5. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  6. 2018Reward Constrained Policy Optimizationcited by 417
  7. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  8. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
  9. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 143
  10. 2020Conservative Safety Critics for Explorationcited by 113
  11. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  12. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 71
12 more
  1. 2022Penalized Proximal Policy Optimization for Safe Reinforcement Learningcited by 68
  2. 2021Risk-Averse Offline Reinforcement Learningcited by 65
  3. 2019Being Optimistic to Be Conservative: Quickly Learning a CVaR Policycited by 63
  4. 2022Safe Reinforcement Learning via Shielding under Partial Observabilitycited by 56
  5. 2021Tactical Optimism and Pessimism for Deep Reinforcement Learningcited by 55
  6. 2016On the Safety of Machine Learning: Cyber-Physical Systems, Decision Sciences, and Data Productscited by 52
  7. 2022Efficient Risk-Averse Reinforcement Learningcited by 43
  8. 2023Safe Exploration in Reinforcement Learning: A Generalized Formulation and Algorithmscited by 28
  9. 2023Trust Region-Based Safe Distributional Reinforcement Learning for Multiple Constraintscited by 21
  10. 2024Off-Policy Primal-Dual Safe Reinforcement Learningcited by 14
  11. 2020DSAC: Distributional Soft Actor-Critic for Risk-Sensitive Reinforcement Learningcited by 8
  12. 2023OVD-Explorer: Optimism Should Not Be the Sole Pursuit of Exploration in Noisy Environmentscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Off-Policy Safe Reinforcement Learning with Constrained Optimistic Explorationcited by 2
  2. 2026COP-Q: Safety-First Reinforcement Learning for Robot Control via Cholesky-Ordered Projectioncited by 0
  3. 2026Utility-Constrained Policy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Risk-Averse Constrained Reinforcement Learning with Optimized Certainty Equivalents
  2. 2025Optimism as Risk-Seeking in Multi-Agent Reinforcement Learning
  3. 2024Improving Actor-Critic Training with Steerable Action-Value Approximation Errors
  4. 2025Exploration from a Primal-Dual Lens: Value-Incentivized Actor-Critic Methods for Sample-Efficient Online RL
  5. 2025Risk-sensitive Actor-Critic with Static Spectral Risk Measures for Online and Offline Reinforcement Learning
  6. 2025Safe But Not Sorry: Reducing Over-Conservatism in Safety Critics via Uncertainty-Aware Modulation
  7. 2025Viability of Future Actions: Robust Safety in Reinforcement Learning via Entropy Regularization
  8. 2025Foundations of Safe Online Reinforcement Learning in the Linear Quadratic Regulator: √T-Regret