# Learning Robust Options by Conditional Value at Risk Optimization

arXiv: https://arxiv.org/abs/1905.09191 · HTML: https://mlsanity.com/paper/1905.09191/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 May 2019.

## Built on by (most cited first, 16 of 16)

- [Efficient Risk-Averse Reinforcement Learning](https://mlsanity.com/paper/2205.05138/) (2022)
- [RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agents](https://mlsanity.com/paper/2102.08159/) (2021)
- [RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorization](https://mlsanity.com/paper/2311.01753/) (2023)
- [Risk-Aware Transfer in Reinforcement Learning using Successor Features](https://mlsanity.com/paper/2105.14127/) (2021)
- [Train Hard, Fight Easy: Robust Meta Reinforcement Learning](https://mlsanity.com/paper/2301.11147/) (2023)
- [An Alternative to Variance: Gini Deviation for Risk-averse Policy Gradient](https://mlsanity.com/paper/2307.08873/) (2023)
- [A Simple Yet Effective Strategy to Robustify the Meta Learning Paradigm](https://mlsanity.com/paper/2310.00708/) (2023)
- [Adjustable Robust Reinforcement Learning for Online 3D Bin Packing](https://mlsanity.com/paper/2310.04323/) (2023)
- [Risk-Averse Finetuning of Large Language Models](https://mlsanity.com/paper/2501.06911/) (2025)
- [Risk-Sensitive Policy with Distributional Reinforcement Learning](https://mlsanity.com/paper/2212.14743/) (2022)
- [On the Theory of Risk-Aware Agents: Bridging Actor-Critic and Economics](https://mlsanity.com/paper/2310.19527/) (2023)
- [Provably Efficient CVaR RL in Low-rank MDPs](https://mlsanity.com/paper/2311.11965/) (2023)
- [Extreme Risk Mitigation in Reinforcement Learning using Extreme Value Theory](https://mlsanity.com/paper/2308.13011/) (2023)
- [Bias-Corrected Peaks-Over-Threshold Estimation of the CVaR](https://mlsanity.com/paper/2103.05059/) (2021)
- [Extreme Value Policy Optimization for Safe Reinforcement Learning](https://mlsanity.com/paper/2601.12008/) (2026)
- [Distributionally Robust Safety Verification of Neural Networks via Worst-Case CVaR](https://mlsanity.com/paper/2509.17413/) (2025)

## Abstract (arXiv)

Options are generally learned by using an inaccurate environment model (or simulator), which contains uncertain model parameters. While there are several methods to learn options that are robust against the uncertainty of model parameters, these methods only consider either the worst case or the average (ordinary) case for learning options. This limited consideration of the cases often produces options that do not work well in the unconsidered case. In this paper, we propose a conditional value at risk (CVaR)-based method to learn options that work well in both the average and worst cases. We extend the CVaR-based policy gradient method proposed by Chow and Ghavamzadeh (2014) to deal with robust Markov decision processes and then apply the extended method to learning robust options. We conduct experiments to evaluate our method in multi-joint robot control tasks (HopperIceBlock, Half-Cheetah, and Walker2D). Experimental results show that our method produces options that 1) give better worst-case performance than the options learned only to minimize the average-case loss, and 2) give better average-case performance than the options learned only to minimize the worst-case loss.