mlsanityData to 7 Oct 2026

Train Hard, Fight Easy: Robust Meta Reinforcement Learning

arXiv 2301.11147 ↗First posted 26 Jan 2023cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

A major challenge of reinforcement learning (RL) in real-world applications is the variation between environments, tasks or clients. Meta-RL (MRL) addresses this issue by learning a meta-policy that adapts to new tasks. Standard MRL methods optimize the average return over tasks, but often suffer from poor results in tasks of high risk or difficulty. This limits system reliability since test tasks are not known in advance. In this work, we define a robust MRL objective with a controlled robustness level. Optimization of analogous robust objectives in RL is known to lead to both *biased gradients* and *data inefficiency*. We prove that the gradient bias disappears in our proposed MRL framework. The data inefficiency is addressed via the novel Robust Meta RL algorithm (RoML). RoML is a meta-algorithm that generates a robust version of any given MRL algorithm, by identifying and over-sampling harder tasks throughout training. We demonstrate that RoML achieves robust returns on multiple navigation and continuous control benchmarks.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 34 of its 55 references that are in the map. First posted on arXiv 26 Jan 2023.

Builds on

34 of 55 references.

References · 34

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  4. 2017Robust Adversarial Reinforcement Learningcited by 439
  5. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  6. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  7. 2019Reinforcement Learning in Healthcare: A Surveycited by 308
  8. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 301
  9. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 270
  10. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
  11. 2020Learned Initializations for Optimizing Coordinate-Based Neural Representationscited by 240
  12. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
22 more
  1. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 162
  2. 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 136
  3. 2018Meta-Reinforcement Learning of Structured Exploration Strategiescited by 128
  4. 2021Replay-Guided Adversarial Environment Designcited by 108
  5. 2014Optimizing the CVaR via Samplingcited by 98
  6. 2021Offline Meta-Reinforcement Learning with Online Self-Supervisioncited by 52
  7. 2020Self-Paced Deep Reinforcement Learningcited by 49
  8. 2022Efficient Risk-Averse Reinforcement Learningcited by 41
  9. 2019A Bayesian Approach to Robust Reinforcement Learningcited by 32
  10. 2020Model-based Adversarial Meta-Reinforcement Learningcited by 32
  11. 2021Self-Paced Context Evaluation for Contextual Reinforcement Learningcited by 25
  12. 2022Controlling Commercial Cooling Systems Using Reinforcement Learningcited by 25
  13. 2020Meta-Reinforcement Learning Robust to Distributional Shift via Model Identification and Experience Relabelingcited by 24
  14. 2021On the Convergence and Optimality of Policy Gradient for Markov Coherent Riskcited by 19
  15. 2019Learning Robust Options by Conditional Value at Risk Optimizationcited by 16
  16. 2021Model-based Meta Reinforcement Learning using Graph Structured Surrogate Modelscited by 16
  17. 2022Distributionally Adaptive Meta Reinforcement Learningcited by 11
  18. 2020Detecting Rewards Deterioration in Episodic Reinforcement Learningcited by 8
  19. 2021ProtoTransformer: A Meta-Learning Approach to Providing Student Feedbackcited by 8
  20. 2022Dynamic Planning in Open-Ended Dialogue using Reinforcement Learningcited by 8
  21. 2021CEM-GD: Cross-Entropy Method with Gradient Descent Planner for Model-Based Reinforcement Learningcited by 6
  22. 2021Automatic Risk Adaptation in Distributional Reinforcement Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 3.

11 papers.

  1. 2025Model Predictive Task Sampling for Efficient and Robust Adaptationcited by 20
  2. 2025Fast and Robust: Task Sampling with Posterior and Diversity Synergies for Adaptive Decision-Makers in Randomized Environmentscited by 11
  3. 2024Certifiably Robust Policies for Uncertain Parametric Environmentscited by 6
  4. 2025Task-Aware Virtual Training: Enhancing Generalization in Meta-Reinforcement Learning for Out-of-Distribution Taskscited by 3
  5. 2026Robust Generalization with Adaptive Optimal Transport Priors for Decision-Focused Learningcited by 2
  6. 2025Bayesian Meta-Reinforcement Learning with Laplace Variational Recurrent Networkscited by 1
  7. 2024Neuromodulated Meta-Learningcited by 0
  8. 2025DIAL: Distribution-Informed Adaptive Learning of Multi-Task Constraints for Safety-Critical Systemscited by 0
  9. 2025Your Offline Policy is Not Trustworthy: Bilevel Reinforcement Learning for Sequential Portfolio Optimizationcited by 0
  10. 2026Probabilistic Performance Guarantees for Multi-Task Reinforcement Learningcited by 0
  11. 2026Disturbance-Aware Aerial Robotics for Ethical Wildlife Monitoringcited by 0