# What Can AutoML Do For Continual Learning?

arXiv: https://arxiv.org/abs/2311.11963 · HTML: https://mlsanity.com/paper/2311.11963/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 29 references that are in the map.
- First posted on arXiv 20 Nov 2023.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [Class-incremental learning: survey and performance evaluation on image classification](https://mlsanity.com/paper/2010.15277/) (2020)
- [AutoML: A Survey of the State-of-the-Art](https://mlsanity.com/paper/1908.00709/) (2019)
- [Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1904.00310/) (2019)
- [Curriculum Learning: A Survey](https://mlsanity.com/paper/2101.10382/) (2021)
- [Overcoming Catastrophic Forgetting by Incremental Moment Matching](https://mlsanity.com/paper/1703.08475/) (2017)
- [Neural Architecture Search: Insights from 1000 Papers](https://mlsanity.com/paper/2301.08727/) (2023)
- [Hyper-Parameter Optimization: A Review of Algorithms and Applications](https://mlsanity.com/paper/2003.05689/) (2020)
- [Efficient Continual Learning with Modular Networks and Task-Driven Priors](https://mlsanity.com/paper/2012.12631/) (2020)
- [Meta-Learning of Neural Architectures for Few-Shot Learning](https://mlsanity.com/paper/1911.11090/) (2019)
- [Architecture Matters in Continual Learning](https://mlsanity.com/paper/2202.00275/) (2022)
- [Online Hyperparameter Optimization for Class-Incremental Learning](https://mlsanity.com/paper/2301.05032/) (2023)
- [Efficient Architecture Search for Continual Learning](https://mlsanity.com/paper/2006.04027/) (2020)
- [The Effect of Task Ordering in Continual Learning](https://mlsanity.com/paper/2205.13323/) (2022)
- [Task Difficulty Aware Parameter Allocation & Regularization for Lifelong Learning](https://mlsanity.com/paper/2304.05288/) (2023)
- [AdaXpert: Adapting Neural Architecture for Growing Data](https://mlsanity.com/paper/2107.00254/) (2021)
- [AdaCL:Adaptive Continual Learning](https://mlsanity.com/paper/2303.13113/) (2023)

## Built on by (most cited first, 1 of 1)

- [Hyperparameter Selection in Continual Learning](https://mlsanity.com/paper/2404.06466/) (2024)

## Abstract (arXiv)

This position paper outlines the potential of AutoML for incremental (continual) learning to encourage more research in this direction. Incremental learning involves incorporating new data from a stream of tasks and distributions to learn enhanced deep representations and adapt better to new tasks. However, a significant limitation of incremental learners is that most current techniques freeze the backbone architecture, hyperparameters, and the order & structure of the learning tasks throughout the learning and adaptation process. We strongly believe that AutoML offers promising solutions to address these limitations, enabling incremental learning to adapt to more diverse real-world tasks. Therefore, instead of directly proposing a new method, this paper takes a step back by posing the question: "What can AutoML do for incremental learning?" We outline three key areas of research that can contribute to making incremental learners more dynamic, highlighting concrete opportunities to apply AutoML methods in novel ways as well as entirely new challenges for AutoML research.