mlsanityData to 14 Oct 2026

Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparator

arXiv 2410.09728 ↗First posted 13 Oct 2024cs.LGcited by 2

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Meta-reinforcement learning (Meta-RL) has attracted attention due to its capability to enhance reinforcement learning (RL) algorithms, in terms of data efficiency and generalizability. In this paper, we develop a bilevel optimization framework for meta-RL (BO-MRL) to learn the meta-prior for task-specific policy adaptation, which implements multiple-step policy optimization on one-time data collection. Beyond existing meta-RL analyses, we provide upper bounds of the expected optimality gap over the task distribution. This metric measures the distance of the policy adaptation from the learned meta-prior to the task-specific optimum, and quantifies the model's generalizability to the task distribution. We empirically validate the correctness of the derived upper bounds and demonstrate the superior effectiveness of the proposed algorithm over benchmarks.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 45 of its 66 references that are in the map. First posted on arXiv 13 Oct 2024.

Builds on

45 of 66 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024A CMDP-within-online framework for Meta-Safe Reinforcement Learning26 May

Other references · 44

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2017Constrained Policy Optimizationcited by 983
  5. 2020Meta-Learning in Neural Networks: A Surveycited by 950
  6. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  7. 2019Meta-Learning with Implicit Gradientscited by 525
  8. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  9. 2019Meta-Learning with Differentiable Convex Optimizationcited by 461
  10. 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 405
  11. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
  12. 2016Learning to reinforcement learncited by 377
32 more
  1. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 276
  2. 2018Approximation Methods for Bilevel Programmingcited by 236
  3. 2017Forward and Reverse Gradient-Based Hyperparameter Optimizationcited by 234
  4. 2016Hyperparameter optimization with approximate gradientcited by 233
  5. 2020Bilevel Optimization: Convergence Analysis and Enhanced Designcited by 222
  6. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 217
  7. 2018Truncated Back-propagation for Bilevel Optimizationcited by 190
  8. 2020On the Iteration Complexity of Hypergradient Computationcited by 155
  9. 2020A Two-Timescale Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Criticcited by 144
  10. 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 134
  11. 2019On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning Algorithmscited by 132
  12. 2020A Finite Time Analysis of Two Time-Scale Actor Critic Methodscited by 122
  13. 2016On Differentiating Parameterized Argmin and Argmax Problems with Application to Bi-level Optimizationcited by 112
  14. 2023A Tutorial on Meta-Reinforcement Learningcited by 108
  15. 2020A Survey of Deep Meta-Learningcited by 100
  16. 2018ProMP: Proximal Meta-Policy Searchcited by 91
  17. 2019Provable Guarantees for Gradient-Based Meta-Learningcited by 67
  18. 2017Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithmcited by 65
  19. 2019Learning-to-Learn Stochastic Gradient Descent with Biased Regularizationcited by 49
  20. 2019Meta Reinforcement Learning for Sim-to-real Domain Adaptationcited by 49
  21. 2018Some Considerations on Learning to Explore via Meta-Reinforcement Learningcited by 44
  22. 2020Rapidly Adaptable Legged Robots via Evolutionary Meta-Learningcited by 43
  23. 2021Generalization of Model-Agnostic Meta-Learning Algorithms: Recurring and Unseen Taskscited by 36
  24. 2020Model-Based Meta-Reinforcement Learning for Flight with Suspended Payloadscited by 30
  25. 2019Guided Meta-Policy Searchcited by 26
  26. 2020On the Global Optimality of Model-Agnostic Meta-Learningcited by 21
  27. 2020On the Convergence Theory of Debiased Model-Agnostic Meta-Reinforcement Learningcited by 19
  28. 2020Safe Active Dynamics Learning and Control: A Sequential Exploration-Exploitation Frameworkcited by 19
  29. 2023Efficient Gradient Approximation Method for Constrained Bilevel Optimizationcited by 15
  30. 2022Provable Generalization of Overparameterized Meta-learning Trained with SGDcited by 9
  31. 2021On the Practical Consistency of Meta-Reinforcement Learning Algorithmscited by 6
  32. 2021Biased Gradient Estimate with Drastic Variance Reduction for Meta Reinforcement Learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2026Safe Continual Reinforcement Learning Methods for Nonstationary Environments. Towards a Survey of the State of the Artcited by 2
  2. 2025Explainable reinforcement learning from human feedback to improve alignmentcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Theoretical Analysis of Meta Reinforcement Learning: Generalization Bounds and Convergence Guarantees
  2. 2024MAMBA: an Effective World Model Approach for Meta-Reinforcement Learning
  3. 2022Discovered Policy Optimisation
  4. 2025Scalable Multi-Objective and Meta Reinforcement Learning via Gradient Estimation
  5. 2025How Should We Meta-Learn Reinforcement Learning Algorithms?
  6. 2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers
  7. 2023Stepsize Learning for Policy Gradient Methods in Contextual Markov Decision Processes
  8. 2026Meta-Multi-Agent Reinforcement Learning for Fast Adaptation of Interactive Policies with Applications to Autonomous Driving