# Train Hard, Fight Easy: Robust Meta Reinforcement Learning

arXiv: https://arxiv.org/abs/2301.11147 · HTML: https://mlsanity.com/paper/2301.11147/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 55 references that are in the map.
- First posted on arXiv 26 Jan 2023.

## Builds on (most cited first, 30 of 34)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [Implicit Quantile Networks for Distributional Reinforcement Learning](https://mlsanity.com/paper/1806.06923/) (2018)
- [Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1610.03295/) (2016)
- [Reinforcement Learning in Healthcare: A Survey](https://mlsanity.com/paper/1908.08796/) (2019)
- [Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning](https://mlsanity.com/paper/1803.11347/) (2018)
- [Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design](https://mlsanity.com/paper/2012.02096/) (2020)
- [Learned Initializations for Optimizing Coordinate-Based Neural Representations](https://mlsanity.com/paper/2012.02189/) (2020)
- [VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning](https://mlsanity.com/paper/1910.08348/) (2019)
- [Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach](https://mlsanity.com/paper/1506.02188/) (2015)
- [EPOpt: Learning Robust Neural Network Policies Using Model Ensembles](https://mlsanity.com/paper/1610.01283/) (2016)
- [Meta-Reinforcement Learning of Structured Exploration Strategies](https://mlsanity.com/paper/1802.07245/) (2018)
- [Replay-Guided Adversarial Environment Design](https://mlsanity.com/paper/2110.02439/) (2021)
- [Optimizing the CVaR via Sampling](https://mlsanity.com/paper/1404.3862/) (2014)
- [Offline Meta-Reinforcement Learning with Online Self-Supervision](https://mlsanity.com/paper/2107.03974/) (2021)
- [Self-Paced Deep Reinforcement Learning](https://mlsanity.com/paper/2004.11812/) (2020)
- [Efficient Risk-Averse Reinforcement Learning](https://mlsanity.com/paper/2205.05138/) (2022)
- [A Bayesian Approach to Robust Reinforcement Learning](https://mlsanity.com/paper/1905.08188/) (2019)
- [Model-based Adversarial Meta-Reinforcement Learning](https://mlsanity.com/paper/2006.08875/) (2020)
- [Controlling Commercial Cooling Systems Using Reinforcement Learning](https://mlsanity.com/paper/2211.07357/) (2022)
- [Meta-Reinforcement Learning Robust to Distributional Shift via Model Identification and Experience Relabeling](https://mlsanity.com/paper/2006.07178/) (2020)
- [Self-Paced Context Evaluation for Contextual Reinforcement Learning](https://mlsanity.com/paper/2106.05110/) (2021)
- [On the Convergence and Optimality of Policy Gradient for Markov Coherent Risk](https://mlsanity.com/paper/2103.02827/) (2021)
- [Model-based Meta Reinforcement Learning using Graph Structured Surrogate Models](https://mlsanity.com/paper/2102.08291/) (2021)
- [Learning Robust Options by Conditional Value at Risk Optimization](https://mlsanity.com/paper/1905.09191/) (2019)
- [Distributionally Adaptive Meta Reinforcement Learning](https://mlsanity.com/paper/2210.03104/) (2022)
- [Detecting Rewards Deterioration in Episodic Reinforcement Learning](https://mlsanity.com/paper/2010.11660/) (2020)

## Built on by (most cited first, 12 of 12)

- [Model Predictive Task Sampling for Efficient and Robust Adaptation](https://mlsanity.com/paper/2501.11039/) (2025)
- [Fast and Robust: Task Sampling with Posterior and Diversity Synergies for Adaptive Decision-Makers in Randomized Environments](https://mlsanity.com/paper/2504.19139/) (2025)
- [Certifiably Robust Policies for Uncertain Parametric Environments](https://mlsanity.com/paper/2408.03093/) (2024)
- [Task-Aware Virtual Training: Enhancing Generalization in Meta-Reinforcement Learning for Out-of-Distribution Tasks](https://mlsanity.com/paper/2502.02834/) (2025)
- [Robust Generalization with Adaptive Optimal Transport Priors for Decision-Focused Learning](https://mlsanity.com/paper/2602.01427/) (2026)
- [Bayesian Meta-Reinforcement Learning with Laplace Variational Recurrent Networks](https://mlsanity.com/paper/2505.18591/) (2025)
- [Probabilistic Performance Guarantees for Multi-Task Reinforcement Learning](https://mlsanity.com/paper/2602.02098/) (2026)
- [Neuromodulated Meta-Learning](https://mlsanity.com/paper/2411.06746/) (2024)
- [DIAL: Distribution-Informed Adaptive Learning of Multi-Task Constraints for Safety-Critical Systems](https://mlsanity.com/paper/2501.18086/) (2025)
- [Your Offline Policy is Not Trustworthy: Bilevel Reinforcement Learning for Sequential Portfolio Optimization](https://mlsanity.com/paper/2505.12759/) (2025)
- [Disturbance-Aware Aerial Robotics for Ethical Wildlife Monitoring](https://mlsanity.com/paper/2606.08249/) (2026)
- [Bridging Risk Approximation Gaps in Model Predictive Task Sampling via In-Context Modeling](https://mlsanity.com/paper/2610.14913/) (2026)

## Abstract (arXiv)

A major challenge of reinforcement learning (RL) in real-world applications is the variation between environments, tasks or clients. Meta-RL (MRL) addresses this issue by learning a meta-policy that adapts to new tasks. Standard MRL methods optimize the average return over tasks, but often suffer from poor results in tasks of high risk or difficulty. This limits system reliability since test tasks are not known in advance. In this work, we define a robust MRL objective with a controlled robustness level. Optimization of analogous robust objectives in RL is known to lead to both *biased gradients* and *data inefficiency*. We prove that the gradient bias disappears in our proposed MRL framework. The data inefficiency is addressed via the novel Robust Meta RL algorithm (RoML). RoML is a meta-algorithm that generates a robust version of any given MRL algorithm, by identifying and over-sampling harder tasks throughout training. We demonstrate that RoML achieves robust returns on multiple navigation and continuous control benchmarks.