mlsanityData to 14 Oct 2026

A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learning

arXiv 2112.15400 ↗First posted 31 Dec 2021cs.LG cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

Gradient-based Meta-RL (GMRL) refers to methods that maintain two-level optimisation procedures wherein the outer-loop meta-learner guides the inner-loop gradient-based reinforcement learner to achieve fast adaptations. In this paper, we develop a unified framework that describes variations of GMRL algorithms and points out that existing stochastic meta-gradient estimators adopted by GMRL are actually biased. Such meta-gradient bias comes from two sources: 1) the compositional bias incurred by the two-level problem structure, which has an upper bound of Obig(KαKhatσIn|τ|-0.5big) w.r.t. inner-loop update step K, learning rate α, estimate variance hatσ2In and sample size |τ|, and 2) the multi-step Hessian estimation bias hatDeltaH due to the use of autodiff, which has a polynomial impact Obig((K-1)(hatDeltaH)K-1big) on the meta-gradient bias. We study tabular MDPs empirically and offer quantitative evidence that testifies our theoretical findings on existing stochastic meta-gradient estimators. Furthermore, we conduct experiments on Iterated Prisoner's Dilemma and Atari games to show how other methods such as off-policy learning and low-bias estimator can help fix the gradient bias for GMRL algorithms in general.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 32 of its 50 references that are in the map. First posted on arXiv 31 Dec 2021.

Builds on

32 of 50 references.

References · 32

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  2. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  3. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  4. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
  5. 2016Learning to reinforcement learncited by 377
  6. 2017Forward and Reverse Gradient-Based Hyperparameter Optimizationcited by 234
  7. 2017Learning with Opponent-Learning Awarenesscited by 222
  8. 2018Meta-Gradient Reinforcement Learningcited by 159
  9. 2019On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning Algorithmscited by 132
  10. 2018Meta-Reinforcement Learning of Structured Exploration Strategiescited by 130
  11. 2017Online Learning Rate Adaptation with Hypergradient Descentcited by 118
  12. 2020Learning to Utilize Shaping Rewards: A New Approach of Reward Shapingcited by 114
20 more
  1. 2017Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environmentscited by 108
  2. 2018On Learning Intrinsic Rewards for Policy Gradient Methodscited by 102
  3. 2018ProMP: Proximal Meta-Policy Searchcited by 91
  4. 2020Discovering Reinforcement Learning Algorithmscited by 77
  5. 2019Improving Generalization in Meta Reinforcement Learning using Learned Objectivescited by 68
  6. 2019Meta-Learning via Learned Losscited by 60
  7. 2020Meta-Gradient Reinforcement Learning with an Objective Discovered Onlinecited by 52
  8. 2018Stable Opponent Shaping in Differentiable Gamescited by 49
  9. 2018DiCE: The Infinitely Differentiable Monte-Carlo Estimatorcited by 46
  10. 2019Discovery of Useful Questions as Auxiliary Taskscited by 45
  11. 2020A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learningcited by 41
  12. 2019What Can Learned Intrinsic Rewards Capture?cited by 36
  13. 2020Towards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive Learningcited by 29
  14. 2020Theoretical Convergence of Multi-Step Model-Agnostic Meta-Learningcited by 28
  15. 2021Discovery of Options via Meta-Learned Subgoalscited by 27
  16. 2020On the Convergence Theory of Debiased Model-Agnostic Meta-Reinforcement Learningcited by 19
  17. 2020Online Meta-Critic Learning for Off-Policy Actor-Critic Methodscited by 17
  18. 2019Loaded DiCE: Trading off Bias and Variance in Any-Order Score Function Estimators for Reinforcement Learningcited by 12
  19. 2021Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluationcited by 5
  20. 2021One Step at a Time: Pros and Cons of Multi-Step Meta-Gradient Reinforcement Learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 3; 2024: 3; 2025: 1; 2026: 3.

12 papers.

  1. 2022TorchOpt: An Efficient Library for Differentiable Optimizationcited by 17
  2. 2024Meta-Learning Linear Quadratic Regulators: A Policy Gradient MAML Approach for Model-free LQRcited by 17
  3. 2024EvIL: Evolution Strategies for Generalisable Imitation Learningcited by 8
  4. 2023Particle-based Online Bayesian Samplingcited by 7
  5. 2023GEAR: A GPU-Centric Experience Replay System for Large Reinforcement Learning Modelscited by 5
  6. 2024Meta-Learning Objectives for Preference Optimizationcited by 2
  7. 2025Unsupervised Meta-Testing with Conditional Neural Processes for Hybrid Meta-Reinforcement Learningcited by 2
  8. 2023Meta-World Conditional Neural Processescited by 1
  9. 2026LiveGraph: Active-Structure Neural Re-ranking for Exercise Recommendationcited by 1
  10. 2026Multitask LQG Control: Performance and Generalization Boundscited by 1
  11. 2022Debiasing Meta-Gradient Reinforcement Learning by Learning the Outer Value Functioncited by 0
  12. 2026UB-SMoE: Universally Balanced Sparse Mixture-of-Experts for Resource-adaptive Federated Fine-tuning of Foundation Modelscited by 0