# Learning Robust Options

arXiv: https://arxiv.org/abs/1802.03236 · HTML: https://mlsanity.com/paper/1802.03236/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Feb 2018.

## Built on by (most cited first, 17 of 17)

- [Robust Reinforcement Learning on State Observations with Learned Optimal Adversary](https://mlsanity.com/paper/2101.08452/) (2021)
- [Automated Reinforcement Learning (AutoRL): A Survey and Open Problems](https://mlsanity.com/paper/2201.03916/) (2022)
- [Twice regularized MDPs and the equivalence between robustness and regularization](https://mlsanity.com/paper/2110.06267/) (2021)
- [Policy Gradient for Rectangular Robust Markov Decision Processes](https://mlsanity.com/paper/2301.13589/) (2023)
- [Robust Imitation Learning against Variations in Environment Dynamics](https://mlsanity.com/paper/2206.09314/) (2022)
- [TempoRL: Learning When to Act](https://mlsanity.com/paper/2106.05262/) (2021)
- [Risk-Aware Transfer in Reinforcement Learning using Successor Features](https://mlsanity.com/paper/2105.14127/) (2021)
- [Max-Min Off-Policy Actor-Critic Method Focusing on Worst-Case Robustness to Model Misspecification](https://mlsanity.com/paper/2211.03413/) (2022)
- [Hierarchical reinforcement learning with natural language subgoals](https://mlsanity.com/paper/2309.11564/) (2023)
- [Robust Reinforcement Learning under model misspecification](https://mlsanity.com/paper/2103.15370/) (2021)
- [Improving Controller Generalization with Dimensionless Markov Decision Processes](https://mlsanity.com/paper/2504.10006/) (2025)
- [Learning Robust Controllers Via Probabilistic Model-Based Policy Search](https://mlsanity.com/paper/2110.13576/) (2021)
- [Twice Regularized Markov Decision Processes: The Equivalence between Robustness and Regularization](https://mlsanity.com/paper/2303.06654/) (2023)
- [TRAM: Test-Time Risk Adaptation with Mixture of Agents](https://mlsanity.com/paper/2408.08812/) (2024)
- [On Practical Robust Reinforcement Learning: Practical Uncertainty Set and Double-Agent Algorithm](https://mlsanity.com/paper/2305.06657/) (2023)
- [Solving Robust MDPs through No-Regret Dynamics](https://mlsanity.com/paper/2305.19035/) (2023)
- [DyPNIPP: Predicting Environment Dynamics for RL-based Robust Informative Path Planning](https://mlsanity.com/paper/2410.17186/) (2024)

## Abstract (arXiv)

Robust reinforcement learning aims to produce policies that have strong guarantees even in the face of environments/transition models whose parameters have strong uncertainty. Existing work uses value-based methods and the usual primitive action setting. In this paper, we propose robust methods for learning temporally abstract actions, in the framework of options. We present a Robust Options Policy Iteration (ROPI) algorithm with convergence guarantees, which learns options that are robust to model uncertainty. We utilize ROPI to learn robust options with the Robust Options Deep Q Network (RO-DQN) that solves multiple tasks and mitigates model misspecification due to model uncertainty. We present experimental results which suggest that policy iteration with linear features may have an inherent form of robustness when using coarse feature representations. In addition, we present experimental results which demonstrate that robustness helps policy iteration implemented on top of deep neural networks to generalize over a much broader range of dynamics than non-robust policy iteration.