# Adaptive Hyperparameter Optimization for Continual Learning Scenarios

arXiv: https://arxiv.org/abs/2403.07015 · HTML: https://mlsanity.com/paper/2403.07015/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 32 references that are in the map.
- First posted on arXiv 9 Mar 2024.

## Builds on (most cited first, 19 of 19)

- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [Tune: A Research Platform for Distributed Model Selection and Training](https://mlsanity.com/paper/1807.05118/) (2018)
- [CORe50: a New Dataset and Benchmark for Continuous Object Recognition](https://mlsanity.com/paper/1705.03550/) (2017)
- [Online Continual Learning in Image Classification: An Empirical Survey](https://mlsanity.com/paper/2101.10423/) (2021)
- [Avalanche: an End-to-End Library for Continual Learning](https://mlsanity.com/paper/2104.00405/) (2021)
- [Rethinking Experience Replay: a Bag of Tricks for Continual Learning](https://mlsanity.com/paper/2010.05595/) (2020)
- [Hyperparameter Importance Across Datasets](https://mlsanity.com/paper/1710.04725/) (2017)
- [Distilled Replay: Overcoming Forgetting through Synthetic Samples](https://mlsanity.com/paper/2103.15851/) (2021)
- [Practical Recommendations for Replay-based Continual Learning Methods](https://mlsanity.com/paper/2203.10317/) (2022)
- [Hyperparameter Transfer Across Developer Adjustments](https://mlsanity.com/paper/2010.13117/) (2020)
- [Continual Learning for Predictive Maintenance: Overview and Challenges](https://mlsanity.com/paper/2301.12467/) (2023)
- [Class-Incremental Learning with Repetition](https://mlsanity.com/paper/2301.11396/) (2023)
- [Continual-Learning-as-a-Service (CLaaS): On-Demand Efficient Adaptation of Predictive Models](https://mlsanity.com/paper/2206.06957/) (2022)
- [Renate: A Library for Real-World Continual Learning](https://mlsanity.com/paper/2304.12067/) (2023)

## Built on by (most cited first, 2 of 2)

- [On the Discrimination and Consistency for Exemplar-Free Class Incremental Learning](https://mlsanity.com/paper/2501.15454/) (2025)
- [Hyperparameter Selection in Continual Learning](https://mlsanity.com/paper/2404.06466/) (2024)

## Abstract (arXiv)

Hyperparameter selection in continual learning scenarios is a challenging and underexplored aspect, especially in practical non-stationary environments. Traditional approaches, such as grid searches with held-out validation data from all tasks, are unrealistic for building accurate lifelong learning systems. This paper aims to explore the role of hyperparameter selection in continual learning and the necessity of continually and automatically tuning them according to the complexity of the task at hand. Hence, we propose leveraging the nature of sequence task learning to improve Hyperparameter Optimization efficiency. By using the functional analysis of variance-based techniques, we identify the most crucial hyperparameters that have an impact on performance. We demonstrate empirically that this approach, agnostic to continual scenarios and strategies, allows us to speed up hyperparameters optimization continually across tasks and exhibit robustness even in the face of varying sequential task orders. We believe that our findings can contribute to the advancement of continual learning methodologies towards more efficient, robust and adaptable models for real-world applications.