mlsanityData to 14 Oct 2026

One Step at a Time: Pros and Cons of Multi-Step Meta-Gradient Reinforcement Learning

arXiv 2111.00206 ↗First posted 30 Oct 2021cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Self-tuning algorithms that adapt the learning process online encourage more effective and robust learning. Among all the methods available, meta-gradients have emerged as a promising approach. They leverage the differentiability of the learning rule with respect to some hyper-parameters to adapt them in an online fashion. Although meta-gradients can be accumulated over multiple learning steps to avoid myopic updates, this is rarely used in practice. In this work, we demonstrate that whilst multi-step meta-gradients do provide a better learning signal in expectation, this comes at the cost of a significant increase in variance, hindering performance. In the light of this analysis, we introduce a novel method mixing multiple inner steps that enjoys a more accurate and robust meta-gradient signal, essentially trading off bias and variance in meta-gradient estimation. When applied to the Snake game, the mixing meta-gradient algorithm can cut the variance by a factor of 3 while achieving similar or higher performance.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 20 of its 27 references that are in the map. First posted on arXiv 30 Oct 2021.

Builds on

20 of 27 references.

References · 20

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  2. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  3. 2018On First-Order Meta-Learning Algorithmscited by 1,018
  4. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
  5. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 217
  6. 2018Meta-Gradient Reinforcement Learningcited by 159
  7. 2018Meta-Reinforcement Learning of Structured Exploration Strategiescited by 130
  8. 2018On Learning Intrinsic Rewards for Policy Gradient Methodscited by 102
  9. 2018Understanding and correcting pathologies in the training of learned optimizerscited by 97
  10. 2018Evolved Policy Gradientscited by 88
  11. 2020Discovering Reinforcement Learning Algorithmscited by 77
  12. 2018Understanding Short-Horizon Bias in Stochastic Meta-Optimizationcited by 69
8 more
  1. 2019Improving Generalization in Meta Reinforcement Learning using Learned Objectivescited by 68
  2. 2019Meta-Learning via Learned Losscited by 60
  3. 2020Meta-Gradient Reinforcement Learning with an Objective Discovered Onlinecited by 52
  4. 2019Discovery of Useful Questions as Auxiliary Taskscited by 45
  5. 2020Discount Factor as a Regularizer in Reinforcement Learningcited by 45
  6. 2021Bootstrapped Meta-Learningcited by 45
  7. 2020Online Meta-Critic Learning for Off-Policy Actor-Critic Methodscited by 17
  8. 2019Beyond Exponentially Discounted Sum: Automatic Learning of Return Functioncited by 7

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 3; 2023: 0; 2024: 1.

5 papers.

  1. 2021A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learningcited by 12
  2. 2022An Investigation of the Bias-Variance Tradeoff in Meta-Gradientscited by 5
  3. 2022AlphaSnake: Policy Iteration on a Nondeterministic NP-hard Markov Decision Processcited by 0
  4. 2022Debiasing Meta-Gradient Reinforcement Learning by Learning the Outer Value Functioncited by 0
  5. 2024Segmenting Action-Value Functions Over Time-Scales in SARSA via TD(Delta)cited by 0