mlsanityData to 7 Oct 2026

Being Optimistic to Be Conservative: Quickly Learning a CVaR Policy

arXiv 1911.01546 ↗First posted 5 Nov 2019cs.LG cs.AIcited by 63

Not on the map. Why?

Abstract (arXiv)

While maximizing expected return is the goal in most reinforcement learning approaches, risk-sensitive objectives such as conditional value at risk (CVaR) are more suitable for many high-stakes applications. However, relatively little is known about how to explore to quickly learn policies with good CVaR. In this paper, we present the first algorithm for sample-efficient learning of CVaR-optimal policies in Markov decision processes based on the optimism in the face of uncertainty principle. This method relies on a novel optimistic version of the distributional Bellman operator that moves probability mass from the lower to the upper tail of the return distribution. We prove asymptotic convergence and optimism of this operator for the tabular policy evaluation case. We further demonstrate that our algorithm finds CVaR-optimal policies substantially faster than existing baselines in several simulated environments with discrete and continuous state spaces.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 12; 2023: 16; 2024: 7; 2025: 9; 2026: 6.

63 papers.

Sort:

By area:

  1. 2021Conservative Offline Distributional Reinforcement Learningcited by 78
  2. 2021Universal Off-Policy Evaluationcited by 43
  3. 2022Efficient Risk-Averse Reinforcement Learningcited by 41
  4. 2023Near-Minimax-Optimal Risk-Sensitive Reinforcement Learning with CVaRcited by 32
  5. 2021RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agentscited by 31
  6. 2021Off-Policy Risk Assessment in Contextual Banditscited by 29
  7. 2021Risk-Averse Bayes-Adaptive Reinforcement Learningcited by 28
  8. 2023The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learningcited by 27
  9. 2022Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 26
  10. 2022Conformal Off-Policy Prediction in Contextual Banditscited by 25
  11. 2022Provably Efficient Risk-Sensitive Reinforcement Learning: Iterated CVaR and Worst Pathcited by 21
  12. 2022One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learningcited by 21
38 more
  1. 2021On the Convergence and Optimality of Policy Gradient for Markov Coherent Riskcited by 19
  2. 2023Distributional Offline Policy Evaluation with Predictive Error Guaranteescited by 19
  3. 2023On Dynamic Programming Decompositions of Static Risk Measures in Markov Decision Processescited by 17
  4. 2022Bridging Distributional and Risk-sensitive Reinforcement Learning with Provable Regret Boundscited by 16
  5. 2024A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalentscited by 15
  6. 2021Lyapunov Barrier Policy Optimizationcited by 12
  7. 2022The Sufficiency of Off-Policyness and Soft Clipping: PPO is still Insufficient according to an Off-Policy Measurecited by 12
  8. 2023Risk-Averse Model Uncertainty for Distributionally Robust Safe Reinforcement Learningcited by 12
  9. 2025Value Flowscited by 12
  10. 2021Two steps to risk sensitivitycited by 8
  11. 2022Enforcing Delayed-Impact Fairness Guaranteescited by 7
  12. 2022Off-Policy Risk Assessment in Markov Decision Processescited by 7
  13. 2023OVD-Explorer: Optimism Should Not Be the Sole Pursuit of Exploration in Noisy Environmentscited by 7
  14. 2024A Simple Mixture Policy Parameterization for Improving Sample Efficiency of CVaR Optimizationcited by 7
  15. 2021Motion Planning for Autonomous Vehicles in the Presence of Uncertainty Using Reinforcement Learningcited by 6
  16. 2024Q-learning for Quantile MDPs: A Decomposition, Performance, and Convergence Analysiscited by 6
  17. 2025Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional Reinforcement Learningcited by 6
  18. 2021Automatic Risk Adaptation in Distributional Reinforcement Learningcited by 5
  19. 2021Planning for Risk-Aversion and Expected Value in MDPscited by 5
  20. 2023Value-Distributional Model-Based Reinforcement Learningcited by 5
  21. 2025Return Capping: Sample-Efficient CVaR Policy Gradient Optimisationcited by 5
  22. 2022Risk-aware Stochastic Shortest Pathcited by 4
  23. 2022Risk-Driven Design of Perception Systemscited by 4
  24. 2021Bias-Corrected Peaks-Over-Threshold Estimation of the CVaRcited by 3
  25. 2022Distributional Actor-Critic Ensemble for Uncertainty-Aware Continuous Controlcited by 3
  26. 2023Pitfall of Optimism: Distributional Reinforcement Learning by Randomizing Risk Criterioncited by 3
  27. 2024Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximationcited by 3
  28. 2025Optimistic Exploration for Risk-Averse Constrained Reinforcement Learningcited by 3
  29. 2021ACReL: Adversarial Conditional value-at-risk Reinforcement Learningcited by 2
  30. 2023A Distribution Optimization Framework for Confidence Bounds of Risk Measurescited by 2
  31. 2023Is Risk-Sensitive Reinforcement Learning Properly Resolved?cited by 2
  32. 2023Distributional Off-Policy Evaluation for Slate Recommendationscited by 2
  33. 2024A unified uncertainty-aware exploration: Combining epistemic and aleatory uncertaintycited by 2
  34. 2024Robust Risk-Sensitive Reinforcement Learning with Conditional Value-at-Riskcited by 2
  35. 2024Burning RED: Unlocking Subtask-Driven Reinforcement Learning and Risk-Awareness in Average-Reward Markov Decision Processescited by 2
  36. 2025Overcoming Non-stationary Dynamics with Evidential Proximal Policy Optimizationcited by 2
  37. 2025Risk-sensitive Actor-Critic with Static Spectral Risk Measures for Online and Offline Reinforcement Learningcited by 2
  38. 2023Toward Risk-based Optimistic Exploration for Cooperative Multi-Agent Reinforcement Learningcited by 1

50 most cited of 63 All 63 (JSON)