# On the Convergence Theory of Debiased Model-Agnostic Meta-Reinforcement Learning

arXiv: https://arxiv.org/abs/2002.05135 · HTML: https://mlsanity.com/paper/2002.05135/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Feb 2020.

## Built on by (most cited first, 19 of 19)

- [A Single-Timescale Analysis For Stochastic Approximation With Multiple Coupled Sequences](https://mlsanity.com/paper/2206.10414/) (2022)
- [Meta-Learning Linear Quadratic Regulators: A Policy Gradient MAML Approach for Model-free LQR](https://mlsanity.com/paper/2401.14534/) (2024)
- [A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learning](https://mlsanity.com/paper/2112.15400/) (2021)
- [A CMDP-within-online framework for Meta-Safe Reinforcement Learning](https://mlsanity.com/paper/2405.16601/) (2024)
- [Meta Stackelberg Game: Robust Federated Learning against Adaptive and Mixed Poisoning Attacks](https://mlsanity.com/paper/2410.17431/) (2024)
- [Sampling Attacks on Meta Reinforcement Learning: A Minimax Formulation and Complexity Analysis](https://mlsanity.com/paper/2208.00081/) (2022)
- [PersA-FL: Personalized Asynchronous Federated Learning](https://mlsanity.com/paper/2210.01176/) (2022)
- [Convergence of Gradient-based MAML in LQR](https://mlsanity.com/paper/2309.06588/) (2023)
- [In-Context Reinforcement Learning via Communicative World Models](https://mlsanity.com/paper/2508.06659/) (2025)
- [Reinforcement Learning with Physics-Informed Symbolic Program Priors for Zero-Shot Wireless Indoor Navigation](https://mlsanity.com/paper/2506.22365/) (2025)
- [Differentiable Information Enhanced Model-Based Reinforcement Learning](https://mlsanity.com/paper/2503.01178/) (2025)
- [On First-Order Meta-Reinforcement Learning with Moreau Envelopes](https://mlsanity.com/paper/2305.12216/) (2023)
- [A Moreau Envelope Approach for LQR Meta-Policy Estimation](https://mlsanity.com/paper/2403.17364/) (2024)
- [Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparator](https://mlsanity.com/paper/2410.09728/) (2024)
- [cMALC-D: Contextual Multi-Agent LLM-Guided Curriculum Learning with Diversity-Based Context Blending](https://mlsanity.com/paper/2508.20818/) (2025)
- [Reinforcement Learning with Brain-Inspired Modulation can Improve Adaptation to Environmental Changes](https://mlsanity.com/paper/2205.09729/) (2022)
- [Latent Guided Sampling for Combinatorial Optimization](https://mlsanity.com/paper/2506.03672/) (2025)
- [Domain-Generalization to Improve Learning in Meta-Learning Algorithms](https://mlsanity.com/paper/2508.09418/) (2025)
- [Personalized Federated Reinforcement Learning via Model-Agnostic Meta-Learning: Convergence of Exact and Hessian-Free Meta-Policy Gradients](https://mlsanity.com/paper/2609.22833/) (2026)

## Abstract (arXiv)

We consider Model-Agnostic Meta-Learning (MAML) methods for Reinforcement Learning (RL) problems, where the goal is to find a policy using data from several tasks represented by Markov Decision Processes (MDPs) that can be updated by one step of stochastic policy gradient for the realized MDP. In particular, using stochastic gradients in MAML update steps is crucial for RL problems since computation of exact gradients requires access to a large number of possible trajectories. For this formulation, we propose a variant of the MAML method, named Stochastic Gradient Meta-Reinforcement Learning (SG-MRL), and study its convergence properties. We derive the iteration and sample complexity of SG-MRL to find an ε-first-order stationary point, which, to the best of our knowledge, provides the first convergence guarantee for model-agnostic meta-reinforcement learning algorithms. We further show how our results extend to the case where more than one step of stochastic policy gradient method is used at test time. Finally, we empirically compare SG-MRL and MAML in several deep RL environments.