mlsanityData to 14 Oct 2026

DRL-ORA: Distributional Reinforcement Learning with Online Risk Adaption

arXiv 2310.05179 ↗First posted 8 Oct 2023cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

One of the main challenges in reinforcement learning (RL) is that the agent has to make decisions that would influence the future performance without having complete knowledge of the environment. Dynamically adjusting the level of epistemic risk during the learning process can help to achieve reliable policies in safety-critical settings with better efficiency. In this work, we propose a new framework, Distributional RL with Online Risk Adaptation (DRL-ORA). This framework quantifies both epistemic and implicit aleatory uncertainties in a unified manner and dynamically adjusts the epistemic risk levels by solving a total variation minimization problem online. The framework unifies the existing variants of risk adaption approaches and offers better explainability and flexibility. The selection of risk levels is performed efficiently via a grid search using a Follow-The-Leader-type algorithm, where the offline oracle also corresponds to a ''satisficing measure'' under a specially modified loss function. We show that DRL-ORA outperforms existing methods that rely on fixed risk levels or manually designed risk level adaptation in multiple classes of tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 59 references that are in the map. First posted on arXiv 8 Oct 2023.

Builds on

31 of 59 references.

References · 31

  1. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  2. 2016Deep Exploration via Bootstrapped DQNcited by 527
  3. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  4. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  5. 2018Distributed Distributional Deterministic Policy Gradientscited by 199
  6. 2019Fully Parameterized Quantile Function for Distributional Reinforcement Learningcited by 88
  7. 2019Estimating Risk and Uncertainty in Deep Reinforcement Learningcited by 70
  8. 2018Benchmarking Reinforcement Learning Algorithms on Real-World Robotscited by 65
  9. 2019Statistics and Samples in Distributional Reinforcement Learningcited by 63
  10. 2019Being Optimistic to Be Conservative: Quickly Learning a CVaR Policycited by 63
  11. 2019Distributional Reinforcement Learning for Efficient Explorationcited by 60
  12. 2021Tactical Optimism and Pessimism for Deep Reinforcement Learningcited by 55
19 more
  1. 2020OR-Gym: A Reinforcement Learning Library for Operations Research Problemscited by 51
  2. 2020Cautious Adaptation For Reinforcement Learning in Safety-Critical Settingscited by 42
  3. 2015Cumulative Prospect Theory Meets Reinforcement Learning: Prediction and Controlcited by 34
  4. 2020Improving Robustness via Risk Averse Distributional Reinforcement Learningcited by 28
  5. 2022Disentangling Epistemic and Aleatoric Uncertainty in Reinforcement Learningcited by 27
  6. 2018Learning in Non-convex Games with an Optimization Oraclecited by 24
  7. 2019ORL: Reinforcement Learning Benchmarks for Online Stochastic Optimization Problemscited by 21
  8. 2019Epistemic Risk-Sensitive Reinforcement Learningcited by 17
  9. 2021SENTINEL: Taming Uncertainty with Ensemble-based Distributional Reinforcement Learningcited by 15
  10. 2021Risk-Conditioned Distributional Soft Actor-Critic for Risk-Sensitive Navigationcited by 15
  11. 2018Exploration by Distributional Reinforcement Learningcited by 13
  12. 2018QUOTA: The Quantile Option Architecture for Reinforcement Learningcited by 13
  13. 2021Addressing Inherent Uncertainty: Risk-Sensitive Behavior Generation for Automated Driving using Distributional Reinforcement Learningcited by 13
  14. 2023Efficient Exploration via Epistemic-Risk-Seeking Policy Optimizationcited by 10
  15. 2021Automatic Risk Adaptation in Distributional Reinforcement Learningcited by 6
  16. 2022Adaptive Risk-Tendency: Nano Drone Navigation in Cluttered Environments with Distributional Reinforcement Learningcited by 6
  17. 2022Risk Perspective Exploration in Distributional Reinforcement Learningcited by 5
  18. 2022RASR: Risk-Averse Soft-Robust MDPs with EVaR and Entropic Riskcited by 4
  19. 2018Data-driven satisficing measure and rankingcited by 1

Built on by

Not cited yet.