mlsanityData to 7 Oct 2026

The Effect of Task Ordering in Continual Learning

arXiv 2205.13323 ↗First posted 26 May 2022cs.LGcited by 30

Not on the map. Why?

Abstract (arXiv)

We investigate the effect of task ordering on continual learning performance. We conduct an extensive series of empirical experiments on synthetic and naturalistic datasets and show that reordering tasks significantly affects the amount of catastrophic forgetting. Connecting to the field of curriculum learning, we show that the effect of task ordering can be exploited to modify continual learning performance, and present a simple approach for doing so. Our method computes the distance between all pairs of tasks, where distance is defined as the source task curvature of a gradient step toward the target task. Using statistically rigorous methods and sound experimental design, we show that task ordering is an important aspect of continual learning that can be modified for improved performance.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 30 of its 53 references that are in the map. First posted on arXiv 26 May 2022.

Builds on

30 of 53 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  4. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  5. 2015Prioritized Experience Replaycited by 1,070
  6. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  7. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  8. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  9. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 359
  10. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 323
  11. 2019On The Power of Curriculum Learning in Training Deep Networkscited by 285
  12. 2017Three Factors Influencing Minima in SGDcited by 270
18 more
  1. 2017Automated Curriculum Learning for Neural Networkscited by 251
  2. 2019Meta-Learning Representations for Continual Learningcited by 189
  3. 2020Understanding the Role of Training Regimes in Continual Learningcited by 177
  4. 2020Using Hindsight to Anchor Past Knowledge in Continual Learningcited by 160
  5. 2020Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semanticscited by 158
  6. 2018Towards Robust Evaluations of Continual Learningcited by 148
  7. 2021Deep Learning Through the Lens of Example Difficultycited by 139
  8. 2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
  9. 2018Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networkscited by 113
  10. 2020When Do Curricula Work?cited by 105
  11. 2020Estimating Example Difficulty Using Variance of Gradientscited by 96
  12. 2014Stochastic Optimization with Importance Samplingcited by 91
  13. 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 86
  14. 2020Learning to Continually Learncited by 82
  15. 2019Toward Understanding Catastrophic Forgetting in Continual Learningcited by 67
  16. 2014Curriculum Learning of Multiple Taskscited by 65
  17. 2016Learning the Curriculum with Bayesian Optimization for Task-Specific Word Representation Learningcited by 22
  18. 2021Behavioral Experiments for Understanding Catastrophic Forgettingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 2; 2024: 7; 2025: 9; 2026: 10.

30 papers.

Sort:

By area:

  1. 2023Theory on Forgetting and Generalization of Continual Learningcited by 69
  2. 2024LEMoE: Advanced Mixture of Experts Adaptor for Lifelong Model Editing of Large Language Modelscited by 14
  3. 2025Optimal Task Order for Continual Learning of Multiple Taskscited by 14
  4. 2024Order parameters and phase transitions of continual learning in deep neural networkscited by 13
  5. 2022Learning to Learn: How to Continuously Teach Humans and Machinescited by 9
  6. 2025Order-Robust Class Incremental Learning: Graph-Driven Dynamic Similarity Groupingcited by 7
  7. 2025Sequence Transferability and Task Order Selection in Continual Learningcited by 5
  8. 2025RICO: Two Realistic Benchmarks and an In-Depth Analysis for Incremental Learning in Object Detectioncited by 5
  9. 2024SAMCL: Empowering SAM to Continually Learn from Dynamic Domains with Extreme Storage Efficiencycited by 3
  10. 2024Benchmarking Sensitivity of Continual Graph Learning for Skeleton-Based Action Recognitioncited by 2
  11. 2025CLDyB: Towards Dynamic Benchmarking for Continual Learning with Pre-trained Modelscited by 2
  12. 2025Interleaved Multitask Learning with Energy Modulated Learning Progresscited by 2
18 more
  1. 2025The Lie of the Average: How Class Incremental Learning Evaluation Deceives You?cited by 2
  2. 2026DynaMiCS: Fine-tuning LLMs with Performance Constraints using Dynamic Mixturescited by 2
  3. 2022Review learning: Real world validation of privacy preserving continual learning across medical institutionscited by 1
  4. 2023What Can AutoML Do For Continual Learning?cited by 1
  5. 2024Can Similarity-Based Domain-Ordering Reduce Catastrophic Forgetting for Intent Recognition?cited by 1
  6. 2024Algorithm Design for Continual Learning in IoT Networkscited by 1
  7. 2025Distributed Learning and Inference Systems: A Networking Perspectivecited by 1
  8. 2026Mitigating Task-Order Sensitivity and Forgetting via Hierarchical Second-Order Consolidationcited by 1
  9. 2026Mitigating Forgetting in Continual Learning with Selective Gradient Projectioncited by 1
  10. 2026Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learningcited by 1
  11. 2026Same Methods, Different Rankings: Trainable Depth as an Evaluation Variable in Continual Learningcited by 1
  12. 2026PATH-Bench: Path-Dependent Evaluation of Lifelong Agentscited by 1
  13. 2024Evaluating and Improving Continual Learning in Spoken Language Understandingcited by 0
  14. 2025Sequencing to Mitigate Catastrophic Forgetting in Continual Learningcited by 0
  15. 2026Repurposing Adversarial Perturbations for Continual Learning: From Defense to Active Alignmentcited by 0
  16. 2026Offline Multi-agent Continual Cooperation via Skill Partition and Reusecited by 0
  17. 2026TASSO: TAsk-Specific Subspace Optimization for Continual Learning of Vision-Language Modelscited by 0
  18. 2026Neural Succession: A Mesoscopic Theory of Invasion, Coexistence, and Stabilization in Continual Learningcited by 0