# Meta-Gradients in Non-Stationary Environments

arXiv: https://arxiv.org/abs/2209.06159 · HTML: https://mlsanity.com/paper/2209.06159/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 22 references that are in the map.
- First posted on arXiv 13 Sep 2022.

## Builds on (most cited first, 14 of 14)

- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [Gradient-based Hyperparameter Optimization through Reversible Learning](https://mlsanity.com/paper/1502.03492/) (2015)
- [Meta-Gradient Reinforcement Learning](https://mlsanity.com/paper/1805.09801/) (2018)
- [On Learning Intrinsic Rewards for Policy Gradient Methods](https://mlsanity.com/paper/1804.06459/) (2018)
- [Meta-Learning with Warped Gradient Descent](https://mlsanity.com/paper/1909.00025/) (2019)
- [Discovering Reinforcement Learning Algorithms](https://mlsanity.com/paper/2007.08794/) (2020)
- [Automated Reinforcement Learning (AutoRL): A Survey and Open Problems](https://mlsanity.com/paper/2201.03916/) (2022)
- [Improving Generalization in Meta Reinforcement Learning using Learned Objectives](https://mlsanity.com/paper/1910.04098/) (2019)
- [Meta-Gradient Reinforcement Learning with an Objective Discovered Online](https://mlsanity.com/paper/2007.08433/) (2020)
- [Bootstrapped Meta-Learning](https://mlsanity.com/paper/2109.04504/) (2021)
- [Discovery of Options via Meta-Learned Subgoals](https://mlsanity.com/paper/2102.06741/) (2021)
- [Introducing Symmetries to Black Box Meta Reinforcement Learning](https://mlsanity.com/paper/2109.10781/) (2021)
- [A Generalizable Approach to Learning Optimizers](https://mlsanity.com/paper/2106.00958/) (2021)
- [Beyond Exponentially Discounted Sum: Automatic Learning of Return Function](https://mlsanity.com/paper/1905.11591/) (2019)

## Built on by (most cited first, 11 of 11)

- [A Definition of Continual Reinforcement Learning](https://mlsanity.com/paper/2307.11046/) (2023)
- [Resetting the Optimizer in Deep RL: An Empirical Study](https://mlsanity.com/paper/2306.17833/) (2023)
- [Optimistic Meta-Gradients](https://mlsanity.com/paper/2301.03236/) (2023)
- [Clustering-Based Weight Orthogonalization for Stabilizing Deep Reinforcement Learning](https://mlsanity.com/paper/2511.11607/) (2025)
- [Learning to Forget: Continual Learning with Adaptive Weight Decay](https://mlsanity.com/paper/2604.27063/) (2026)
- [Debiasing Meta-Gradient Reinforcement Learning by Learning the Outer Value Function](https://mlsanity.com/paper/2211.10550/) (2022)
- [POMRL: No-Regret Learning-to-Plan with Increasing Horizons](https://mlsanity.com/paper/2212.14530/) (2022)
- [MERMAIDE: Learning to Align Learners using Model-Based Meta-Learning](https://mlsanity.com/paper/2304.04668/) (2023)
- [Acceleration in Policy Optimization](https://mlsanity.com/paper/2306.10587/) (2023)
- [Ever Evolving Evaluator (EV3): Towards Flexible and Reliable Meta-Optimization for Knowledge Distillation](https://mlsanity.com/paper/2310.18893/) (2023)
- [Ergodic Risk Measures: Towards a Risk-Aware Foundation for Continual Reinforcement Learning](https://mlsanity.com/paper/2510.02945/) (2025)

## Abstract (arXiv)

Meta-gradient methods (Xu et al., 2018; Zahavy et al., 2020) offer a promising solution to the problem of hyperparameter selection and adaptation in non-stationary reinforcement learning problems. However, the properties of meta-gradients in such environments have not been systematically studied. In this work, we bring new clarity to meta-gradients in non-stationary environments. Concretely, we ask: (i) how much information should be given to the learned optimizers, so as to enable faster adaptation and generalization over a lifetime, (ii) what meta-optimizer functions are learned in this process, and (iii) whether meta-gradient methods provide a bigger advantage in highly non-stationary environments. To study the effect of information provided to the meta-optimizer, as in recent works (Flennerhag et al., 2021; Almeida et al., 2021), we replace the tuned meta-parameters of fixed update rules with learned meta-parameter functions of selected context features. The context features carry information about agent performance and changes in the environment and hence can inform learned meta-parameter schedules. We find that adding more contextual information is generally beneficial, leading to faster adaptation of meta-parameter values and increased performance over a lifetime. We support these results with a qualitative analysis of resulting meta-parameter schedules and learned functions of context features. Lastly, we find that without context, meta-gradients do not provide a consistent advantage over the baseline in highly non-stationary environments. Our findings suggest that contextualizing meta-gradients can play a pivotal role in extracting high performance from meta-gradients in non-stationary settings.