mlsanityData to 7 Oct 2026

Optimal Task Order for Continual Learning of Multiple Tasks

arXiv 2502.03350 ↗First posted 5 Feb 2025stat.ML cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

Continual learning of multiple tasks remains a major challenge for neural networks. Here, we investigate how task order influences continual learning and propose a strategy for optimizing it. Leveraging a linear teacher-student model with latent factors, we derive an analytical expression relating task similarity and ordering to learning performance. Our analysis reveals two principles that hold under a wide parameter range: (1) tasks should be arranged from the least representative to the most typical, and (2) adjacent tasks should be dissimilar. We validate these rules on both synthetic data and real-world image classification datasets (Fashion-MNIST, CIFAR-10, CIFAR-100), demonstrating consistent performance improvements in both multilayer perceptrons and convolutional neural networks. Our work thus presents a generalizable framework for task-order optimization in task-incremental continual learning.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 38 of its 58 references that are in the map. First posted on arXiv 5 Feb 2025.

Builds on

38 of 58 references.

References · 38

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  4. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  5. 2017Continual Learning with Deep Generative Replaycited by 1,152
  6. 2018Experience Replay for Continual Learningcited by 1,074
  7. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  8. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  9. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  10. 2019Three scenarios for continual learningcited by 610
  11. 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 350
  12. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
26 more
  1. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  2. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 201
  3. 2018A mathematical theory of semantic development in deep neural networkscited by 182
  4. 2020Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semanticscited by 158
  5. 2015Statistical physics of inference: Thresholds and algorithmscited by 128
  6. 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 86
  7. 2021Continual Learning of a Mixed Sequence of Similar and Dissimilar Taskscited by 80
  8. 2022How catastrophic can catastrophic forgetting be in linear regression?cited by 80
  9. 2023Theory on Forgetting and Generalization of Continual Learningcited by 69
  10. 2014Curriculum Learning of Multiple Taskscited by 65
  11. 2018An Overview of Machine Teachingcited by 63
  12. 2022Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfercited by 59
  13. 2022CLAD: A realistic Continual Learning benchmark for Autonomous Drivingcited by 35
  14. 2023The Ideal Continual Learner: An Agent That Never Forgetscited by 34
  15. 2023Continual Learning in Linear Classification on Separable Datacited by 31
  16. 2022The Effect of Task Ordering in Continual Learningcited by 30
  17. 2022Analysis of Catastrophic Forgetting for Random Orthogonal Transformation Tasks in the Overparameterized Regimecited by 28
  18. 2024The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting -- An Analytical Modelcited by 26
  19. 2021Statistical Mechanical Analysis of Catastrophic Forgetting in Continual Learning with Teacher and Student Networkscited by 22
  20. 2023Continual Learning in Medical Image Analysis: A Comprehensive Review of Recent Advancements and Future Prospectscited by 22
  21. 2024Disentangling and Mitigating the Impact of Task Similarity for Continual Learningcited by 19
  22. 2024Optimal Protocols for Continual Learning via Statistical Physics and Control Theorycited by 14
  23. 2022Learning to Learn: How to Continuously Teach Humans and Machinescited by 9
  24. 2022Statistical mechanics of continual learning: variational principle and mean-field potentialcited by 9
  25. 2023Sub-network Discovery and Soft-masking for Continual Learning of Mixed Taskscited by 7
  26. 2024Is Continual Learning Ready for Real-world Challenges?cited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 13.

14 papers.

Sort:

By area:

  1. 2025Statistical Deficiency for Task Inclusion Estimationcited by 1
  2. 2026Provable Effects of Data Replay in Continual Learning: A Feature Learning Perspectivecited by 1
  3. 2026Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learningcited by 1
  4. 2026Interference and Retention in Continual Learningcited by 1
  5. 2026Cascaded Transfer: Learning Many Tasks under Budget Constraintscited by 0
  6. 2026From Order to Distribution: An Exact Operator Framework for Forgetting in Continual Learningcited by 0
  7. 2026Understanding Imbalanced Forgetting in Rehearsal-Based Class-Incremental Learningcited by 0
  8. 2026Understanding Generalization and Forgetting in In-Context Continual Learningcited by 0
  9. 2026Offline Multi-agent Continual Cooperation via Skill Partition and Reusecited by 0
  10. 2026Convergence of Continual Learning in Homogeneous Deep Networkscited by 0
  11. 2026TASSO: TAsk-Specific Subspace Optimization for Continual Learning of Vision-Language Modelscited by 0
  12. 2026The Sequential Price of Continual Learningcited by 0
2 more
  1. 2026Neural Succession: A Mesoscopic Theory of Invasion, Coexistence, and Stabilization in Continual Learningcited by 0
  2. 2026When Is Deletion Ordering Tractable? From Update Dynamics to Permutation Structurecited by 0