# Optimal Task Order for Continual Learning of Multiple Tasks

arXiv: https://arxiv.org/abs/2502.03350 · HTML: https://mlsanity.com/paper/2502.03350/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 58 references that are in the map.
- First posted on arXiv 5 Feb 2025.

## Builds on (most cited first, 30 of 38)

- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://mlsanity.com/paper/2308.08747/) (2023)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](https://mlsanity.com/paper/2003.04960/) (2020)
- [Understanding self-supervised Learning Dynamics without Contrastive Pairs](https://mlsanity.com/paper/2102.06810/) (2021)
- [High-dimensional dynamics of generalization error in neural networks](https://mlsanity.com/paper/1710.03667/) (2017)
- [How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned](https://mlsanity.com/paper/2102.02915/) (2021)
- [A mathematical theory of semantic development in deep neural networks](https://mlsanity.com/paper/1810.10531/) (2018)
- [Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics](https://mlsanity.com/paper/2007.07400/) (2020)
- [Statistical physics of inference: Thresholds and algorithms](https://mlsanity.com/paper/1511.02476/) (2015)
- [Continual Learning in the Teacher-Student Setup: Impact of Task Similarity](https://mlsanity.com/paper/2107.04384/) (2021)
- [Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks](https://mlsanity.com/paper/2112.10017/) (2021)
- [How catastrophic can catastrophic forgetting be in linear regression?](https://mlsanity.com/paper/2205.09588/) (2022)
- [Theory on Forgetting and Generalization of Continual Learning](https://mlsanity.com/paper/2302.05836/) (2023)
- [Curriculum Learning of Multiple Tasks](https://mlsanity.com/paper/1412.1353/) (2014)
- [An Overview of Machine Teaching](https://mlsanity.com/paper/1801.05927/) (2018)
- [Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfer](https://mlsanity.com/paper/2211.00789/) (2022)
- [CLAD: A realistic Continual Learning benchmark for Autonomous Driving](https://mlsanity.com/paper/2210.03482/) (2022)
- [The Ideal Continual Learner: An Agent That Never Forgets](https://mlsanity.com/paper/2305.00316/) (2023)
- [Continual Learning in Linear Classification on Separable Data](https://mlsanity.com/paper/2306.03534/) (2023)
- [The Effect of Task Ordering in Continual Learning](https://mlsanity.com/paper/2205.13323/) (2022)
- [Analysis of Catastrophic Forgetting for Random Orthogonal Transformation Tasks in the Overparameterized Regime](https://mlsanity.com/paper/2207.06475/) (2022)
- [The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting -- An Analytical Model](https://mlsanity.com/paper/2401.12617/) (2024)

## Built on by (most cited first, 14 of 14)

- [Statistical Deficiency for Task Inclusion Estimation](https://mlsanity.com/paper/2503.05491/) (2025)
- [Provable Effects of Data Replay in Continual Learning: A Feature Learning Perspective](https://mlsanity.com/paper/2602.02767/) (2026)
- [Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning](https://mlsanity.com/paper/2604.15414/) (2026)
- [Interference and Retention in Continual Learning](https://mlsanity.com/paper/2607.09202/) (2026)
- [Cascaded Transfer: Learning Many Tasks under Budget Constraints](https://mlsanity.com/paper/2601.21513/) (2026)
- [From Order to Distribution: An Exact Operator Framework for Forgetting in Continual Learning](https://mlsanity.com/paper/2604.13460/) (2026)
- [Understanding Imbalanced Forgetting in Rehearsal-Based Class-Incremental Learning](https://mlsanity.com/paper/2605.14785/) (2026)
- [Understanding Generalization and Forgetting in In-Context Continual Learning](https://mlsanity.com/paper/2605.28705/) (2026)
- [Offline Multi-agent Continual Cooperation via Skill Partition and Reuse](https://mlsanity.com/paper/2606.25389/) (2026)
- [Convergence of Continual Learning in Homogeneous Deep Networks](https://mlsanity.com/paper/2606.30559/) (2026)
- [TASSO: TAsk-Specific Subspace Optimization for Continual Learning of Vision-Language Models](https://mlsanity.com/paper/2608.21487/) (2026)
- [The Sequential Price of Continual Learning](https://mlsanity.com/paper/2609.29674/) (2026)
- [Neural Succession: A Mesoscopic Theory of Invasion, Coexistence, and Stabilization in Continual Learning](https://mlsanity.com/paper/2609.36375/) (2026)
- [When Is Deletion Ordering Tractable? From Update Dynamics to Permutation Structure](https://mlsanity.com/paper/2610.01149/) (2026)

## Abstract (arXiv)

Continual learning of multiple tasks remains a major challenge for neural networks. Here, we investigate how task order influences continual learning and propose a strategy for optimizing it. Leveraging a linear teacher-student model with latent factors, we derive an analytical expression relating task similarity and ordering to learning performance. Our analysis reveals two principles that hold under a wide parameter range: (1) tasks should be arranged from the least representative to the most typical, and (2) adjacent tasks should be dissimilar. We validate these rules on both synthetic data and real-world image classification datasets (Fashion-MNIST, CIFAR-10, CIFAR-100), demonstrating consistent performance improvements in both multilayer perceptrons and convolutional neural networks. Our work thus presents a generalizable framework for task-order optimization in task-incremental continual learning.