# Step-size Optimization for Continual Learning

arXiv: https://arxiv.org/abs/2401.17401 · HTML: https://mlsanity.com/paper/2401.17401/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 30 references that are in the map.
- First posted on arXiv 30 Jan 2024.

## Builds on (most cited first, 12 of 12)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives](https://mlsanity.com/paper/1407.0202/) (2014)
- [Meta-Gradient Reinforcement Learning](https://mlsanity.com/paper/1805.09801/) (2018)
- [Meta-Gradient Reinforcement Learning with an Objective Discovered Online](https://mlsanity.com/paper/2007.08433/) (2020)
- [MetaGrad: Multiple Learning Rates in Online Learning](https://mlsanity.com/paper/1604.08740/) (2016)
- [Natural Neural Networks](https://mlsanity.com/paper/1507.00210/) (2015)
- [WNGrad: Learn the Learning Rate in Gradient Descent](https://mlsanity.com/paper/1803.02865/) (2018)
- [Meta-descent for Online, Continual Prediction](https://mlsanity.com/paper/1907.07751/) (2019)
- [A History of Meta-gradient: Gradient Methods for Meta-learning](https://mlsanity.com/paper/2202.09701/) (2022)
- [Metatrace Actor-Critic: Online Step-size Tuning by Meta-gradient Descent for Reinforcement Learning Control](https://mlsanity.com/paper/1805.04514/) (2018)
- [Learning Feature Relevance Through Step Size Adaptation in Temporal-Difference Learning](https://mlsanity.com/paper/1903.03252/) (2019)

## Built on by (most cited first, 6 of 6)

- [Reward Centering](https://mlsanity.com/paper/2405.09999/) (2024)
- [MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters](https://mlsanity.com/paper/2402.02342/) (2024)
- [Anytime Training with Schedule-Free Spectral Optimization](https://mlsanity.com/paper/2605.23061/) (2026)
- [Learning to Forget: Continual Learning with Adaptive Weight Decay](https://mlsanity.com/paper/2604.27063/) (2026)
- [Position: Deployed Reinforcement Learning should be Continual](https://mlsanity.com/paper/2606.04029/) (2026)
- [Dynamic Learning Rate for Deep Reinforcement Learning: A Bandit Approach](https://mlsanity.com/paper/2410.12598/) (2024)

## Abstract (arXiv)

In continual learning, a learner has to keep learning from the data over its whole life time. A key issue is to decide what knowledge to keep and what knowledge to let go. In a neural network, this can be implemented by using a step-size vector to scale how much gradient samples change network weights. Common algorithms, like RMSProp and Adam, use heuristics, specifically normalization, to adapt this step-size vector. In this paper, we show that those heuristics ignore the effect of their adaptation on the overall objective function, for example by moving the step-size vector away from better step-size vectors. On the other hand, stochastic meta-gradient descent algorithms, like IDBD (Sutton, 1992), explicitly optimize the step-size vector with respect to the overall objective function. On simple problems, we show that IDBD is able to consistently improve step-size vectors, where RMSProp and Adam do not. We explain the differences between the two approaches and their respective limitations. We conclude by suggesting that combining both approaches could be a promising future direction to improve the performance of neural networks in continual learning.