# Towards Robust Evaluations of Continual Learning

arXiv: https://arxiv.org/abs/1805.09733 · HTML: https://mlsanity.com/paper/1805.09733/

## Key facts

- Cited by 148 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 May 2018.

## Built on by (most cited first, 30 of 148)

- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [Gradient Projection Memory for Continual Learning](https://mlsanity.com/paper/2103.09762/) (2021)
- [DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion](https://mlsanity.com/paper/2111.11326/) (2021)
- [Online Continual Learning in Image Classification: An Empirical Survey](https://mlsanity.com/paper/2101.10423/) (2021)
- [New Insights on Reducing Abrupt Representation Change in Online Continual Learning](https://mlsanity.com/paper/2104.05025/) (2021)
- [Class-Incremental Continual Learning into the eXtended DER-verse](https://mlsanity.com/paper/2201.00766/) (2022)
- [Continual World: A Robotic Benchmark For Continual Reinforcement Learning](https://mlsanity.com/paper/2105.10919/) (2021)
- [Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System](https://mlsanity.com/paper/2201.12604/) (2022)
- [The CLEAR Benchmark: Continual LEArning on Real-World Imagery](https://mlsanity.com/paper/2201.06289/) (2022)
- [Continual Learning in the Teacher-Student Setup: Impact of Task Similarity](https://mlsanity.com/paper/2107.04384/) (2021)
- [Online Continual Learning with Natural Distribution Shifts: An Empirical Study with Visual Data](https://mlsanity.com/paper/2108.09020/) (2021)
- [Continual Learning via Local Module Composition](https://mlsanity.com/paper/2111.07736/) (2021)
- [Model Zoo: A Growing "Brain" That Learns Continually](https://mlsanity.com/paper/2106.03027/) (2021)
- [Posterior Meta-Replay for Continual Learning](https://mlsanity.com/paper/2103.01133/) (2021)
- [Transfer without Forgetting](https://mlsanity.com/paper/2206.00388/) (2022)
- [Same State, Different Task: Continual Reinforcement Learning without Interference](https://mlsanity.com/paper/2106.02940/) (2021)
- [Class-Incremental Learning with Generative Classifiers](https://mlsanity.com/paper/2104.10093/) (2021)
- [On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning](https://mlsanity.com/paper/2210.06443/) (2022)
- [Continual Learning for Blind Image Quality Assessment](https://mlsanity.com/paper/2102.09717/) (2021)
- [Distilled Replay: Overcoming Forgetting through Synthetic Samples](https://mlsanity.com/paper/2103.15851/) (2021)
- [Continual Learning for Recurrent Neural Networks: an Empirical Evaluation](https://mlsanity.com/paper/2103.07492/) (2021)
- [Disentangling Transfer in Continual Reinforcement Learning](https://mlsanity.com/paper/2209.13900/) (2022)
- [The Effect of Task Ordering in Continual Learning](https://mlsanity.com/paper/2205.13323/) (2022)
- [Analysis of Catastrophic Forgetting for Random Orthogonal Transformation Tasks in the Overparameterized Regime](https://mlsanity.com/paper/2207.06475/) (2022)
- [Online Continual Learning Without the Storage Constraint](https://mlsanity.com/paper/2305.09253/) (2023)
- [Continual Learning via Sequential Function-Space Variational Inference](https://mlsanity.com/paper/2312.17210/) (2023)
- [CLEVA-Compass: A Continual Learning EValuation Assessment Compass to Promote Research Transparency and Comparability](https://mlsanity.com/paper/2110.03331/) (2021)
- [Continual Learning on Noisy Data Streams via Self-Purified Replay](https://mlsanity.com/paper/2110.07735/) (2021)
- [Error Sensitivity Modulation based Experience Replay: Mitigating Abrupt Representation Drift in Continual Learning](https://mlsanity.com/paper/2302.11344/) (2023)
- [Online-LoRA: Task-free Online Continual Learning via Low Rank Adaptation](https://mlsanity.com/paper/2411.05663/) (2024)

## Abstract (arXiv)

Experiments used in current continual learning research do not faithfully assess fundamental challenges of learning continually. Instead of assessing performance on challenging and representative experiment designs, recent research has focused on increased dataset difficulty, while still using flawed experiment set-ups. We examine standard evaluations and show why these evaluations make some continual learning approaches look better than they are. We introduce desiderata for continual learning evaluations and explain why their absence creates misleading comparisons. Based on our desiderata we then propose new experiment designs which we demonstrate with various continual learning approaches and datasets. Our analysis calls for a reprioritization of research effort by the community.