mlsanityData to 7 Oct 2026

Risk-Averse Offline Reinforcement Learning

arXiv 2102.05371 ↗First posted 10 Feb 2021cs.LGcited by 65

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Training Reinforcement Learning (RL) agents in high-stakes applications might be too prohibitive due to the risk associated to exploration. Thus, the agent can only use data previously collected by safe policies. While previous work considers optimizing the average performance using offline data, we focus on optimizing a risk-averse criteria, namely the CVaR. In particular, we present the Offline Risk-Averse Actor-Critic (O-RAAC), a model-free RL algorithm that is able to learn risk-averse policies in a fully offline setting. We show that O-RAAC learns policies with higher CVaR than risk-neutral approaches in different robot control tasks. Furthermore, considering risk-averse criteria guarantees distributional robustness of the average performance with respect to particular distribution shifts. We demonstrate empirically that in the presence of natural distribution-shifts, O-RAAC learns policies with good average performance.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 29 of its 60 references that are in the map. First posted on arXiv 10 Feb 2021.

Builds on

29 of 60 references.

Closest ancestors · 4

Same direction, newest first

  1. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  2. 2020MOPO: Model-based Offline Policy Optimization27 May
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  4. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb

Other references · 25

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016Generative Adversarial Imitation Learningcited by 1,576
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  5. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  6. 2017Towards Principled Methods for Training Generative Adversarial Networkscited by 460
  7. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  8. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  9. 2019Challenges of Real-World Reinforcement Learningcited by 282
  10. 2019Monte Carlo Gradient Estimation in Machine Learningcited by 281
  11. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  12. 2020Large-Scale Methods for Distributionally Robust Optimizationcited by 179
13 more
  1. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 162
  2. 2016Variance-based regularization with convex objectivescited by 153
  3. 2013Risk-sensitive Reinforcement Learningcited by 129
  4. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  5. 2014Optimizing the CVaR via Samplingcited by 98
  6. 2014Algorithms for CVaR Optimization in MDPscited by 98
  7. 2019Risk Averse Robust Adversarial Reinforcement Learningcited by 54
  8. 2019Worst Cases Policy Gradientscited by 52
  9. 2019Adaptive Sampling for Stochastic Risk-Averse Learningcited by 39
  10. 2015Cumulative Prospect Theory Meets Reinforcement Learning: Prediction and Controlcited by 34
  11. 2020Improving Robustness via Risk Averse Distributional Reinforcement Learningcited by 28
  12. 2018Risk-Sensitive Generative Adversarial Imitation Learningcited by 16
  13. 2017RAIL: Risk-Averse Imitation Learningcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 19; 2023: 12; 2024: 12; 2025: 7; 2026: 8.

65 papers.

Sort:

By area:

  1. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 337
  2. 2021Conservative Offline Distributional Reinforcement Learningcited by 78
  3. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 69
  4. 2022Probable Domain Generalization via Quantile Risk Minimizationcited by 66
  5. 2022Uncertainty Estimation for Language Reward Modelscited by 35
  6. 2023Near-Minimax-Optimal Risk-Sensitive Reinforcement Learning with CVaRcited by 32
  7. 2025Uncertainty-aware Latent Safety Filters for Avoiding Out-of-Distribution Failurescited by 30
  8. 2022One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learningcited by 21
  9. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  10. 2023RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorizationcited by 19
  11. 2023Constrained Policy Optimization with Explicit Behavior Density for Offline Reinforcement Learningcited by 16
  12. 2024A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalentscited by 15
38 more
  1. 2022Distributionally Robust Optimization via Ball Oracle Accelerationcited by 14
  2. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learningcited by 14
  3. 2023Learning Risk-Aware Quadrupedal Locomotion using Distributional Reinforcement Learningcited by 12
  4. 2024Causal Action Influence Aware Counterfactual Data Augmentationcited by 12
  5. 2022A Behavior Regularized Implicit Policy for Offline Reinforcement Learningcited by 11
  6. 2022User-Interactive Offline Reinforcement Learningcited by 11
  7. 2022Comparing Model-free and Model-based Algorithms for Offline Reinforcement Learningcited by 9
  8. 2022Robust Reinforcement Learning with Distributional Risk-averse formulationcited by 9
  9. 2021Two steps to risk sensitivitycited by 8
  10. 2023Decision-Making Under Uncertainty: Beyond Probabilitiescited by 8
  11. 2021Distributional Offline Continuous-Time Reinforcement Learning with Neural Physics-Informed PDEs (SciPhy RL for DOCTR-L)cited by 6
  12. 2022Optimistic Curiosity Exploration and Conservative Exploitation with Linear Reward Shapingcited by 6
  13. 2022Risk-Sensitive Policy with Distributional Reinforcement Learningcited by 6
  14. 2024Risk-Sensitive Soft Actor-Critic for Robust Deep Reinforcement Learning under Distribution Shiftscited by 6
  15. 2026Efficient Tail-Aware Generative Optimization via Flow Model Fine-Tuningcited by 6
  16. 2022Risk-Averse No-Regret Learning in Online Convex Gamescited by 5
  17. 2022Adaptive Risk-Tendency: Nano Drone Navigation in Cluttered Environments with Distributional Reinforcement Learningcited by 5
  18. 2023Extreme Risk Mitigation in Reinforcement Learning using Extreme Value Theorycited by 5
  19. 2024Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learningcited by 5
  20. 2024Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency modelcited by 4
  21. 2021Deep Reinforcement Learning for Equal Risk Pricing and Hedging under Dynamic Expectile Risk Measurescited by 3
  22. 2021Robust Dynamic Bus Control: A Distributional Multi-agent Reinforcement Learning Approachcited by 3
  23. 2021Measuring Data Quality for Dataset Selection in Offline Reinforcement Learningcited by 3
  24. 2022Distributional Actor-Critic Ensemble for Uncertainty-Aware Continuous Controlcited by 3
  25. 2024Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximationcited by 3
  26. 2025Optimistic Exploration for Risk-Averse Constrained Reinforcement Learningcited by 3
  27. 2026Extreme Value Policy Optimization for Safe Reinforcement Learningcited by 3
  28. 2022Tailoring to the Tails: Risk Measures for Fine-Grained Tail Sensitivitycited by 2
  29. 2023Approximate Shielding of Atari Agents for Safe Explorationcited by 2
  30. 2023PACER: A Fully Push-forward-based Distributional Reinforcement Learning Algorithmcited by 2
  31. 2023Is Risk-Sensitive Reinforcement Learning Properly Resolved?cited by 2
  32. 2024UOEP: User-Oriented Exploration Policy for Enhancing Long-Term User Experiences in Recommender Systemscited by 2
  33. 2024Diffusion Policies for Risk-Averse Behavior Modeling in Offline Reinforcement Learningcited by 2
  34. 2024Risk-averse Learning with Non-Stationary Distributionscited by 2
  35. 2025Risk-sensitive Actor-Critic with Static Spectral Risk Measures for Online and Offline Reinforcement Learningcited by 2
  36. 2022Beyond Bayes-optimality: meta-learning what you know you don't knowcited by 1
  37. 2023JoinGym: An Efficient Query Optimization Environment for Reinforcement Learningcited by 1
  38. 2024Risk-Sensitive Multi-Agent Reinforcement Learning in Network Aggregative Markov Gamescited by 1

50 most cited of 65 All 65 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2020Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluation
  2. 2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL
  3. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation
  4. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning
  5. 2025Online Optimization for Offline Safe Reinforcement Learning
  6. 2024Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learning
  7. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning
  8. 2021Offline RL Without Off-Policy Evaluation