mlsanityData to 14 Oct 2026

Continual Learning in the Teacher-Student Setup: Impact of Task Similarity

arXiv 2107.04384 ↗First posted 9 Jul 2021stat.ML cond-mat.stat-mech cs.LGcited by 88

On the map: Knowledge distillation → Distillation for continual learning

Abstract (arXiv)

Continual learning-the ability to learn many tasks in sequence-is critical for artificial learning systems. Yet standard training methods for deep networks often suffer from catastrophic forgetting, where learning new tasks erases knowledge of earlier tasks. While catastrophic forgetting labels the problem, the theoretical reasons for interference between tasks remain unclear. Here, we attempt to narrow this gap between theory and practice by studying continual learning in the teacher-student setup. We extend previous analytical work on two-layer networks in the teacher-student setup to multiple teachers. Using each teacher to represent a different task, we investigate how the relationship between teachers affects the amount of forgetting and transfer exhibited by the student when the task switches. In line with recent work, we find that when tasks depend on similar features, intermediate task similarity leads to greatest forgetting. However, feature similarity is only one way in which tasks may be related. The teacher-student approach allows us to disentangle task similarity at the level of readouts (hidden-to-output weights) and features (input-to-hidden weights). We find a complex interplay between both types of similarity, initial transfer/forgetting rates, maximum transfer/forgetting, and long-term transfer/forgetting. Together, these results help illuminate the diverse factors contributing to catastrophic forgetting.

Key facts. Cited by 88 papers in the map (data through 14 Oct 2026). Placed in Distillation for continual learning, one of 12 directions in Knowledge distillation. Builds on 38 of its 62 references that are in the map. First posted on arXiv 9 Jul 2021.

Builds on

38 of 62 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Statistical Mechanical Analysis of Catastrophic Forgetting in Continual Learning with Teacher and Student Networks16 May

Other references · 37

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  4. 2016Learning without Forgettingcited by 2,496
  5. 2016Progressive Neural Networkscited by 1,361
  6. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  7. 2017Continual Learning with Deep Generative Replaycited by 1,158
  8. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  9. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  10. 2018On Lazy Training in Differentiable Programmingcited by 675
  11. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  12. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
25 more
  1. 2020What is being transferred in transfer learning?cited by 368
  2. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 257
  3. 2017High-dimensional dynamics of generalization error in neural networkscited by 241
  4. 2020Understanding the Role of Training Regimes in Continual Learningcited by 180
  5. 2020Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semanticscited by 161
  6. 2018Towards Robust Evaluations of Continual Learningcited by 148
  7. 2015Statistical physics of inference: Thresholds and algorithmscited by 128
  8. 2017Theoretical insights into the optimization landscape of over-parameterized shallow neural networkscited by 128
  9. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  10. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  11. 2020A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrixcited by 103
  12. 2017Recovery Guarantees for One-hidden-layer Neural Networkscited by 94
  13. 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
  14. 2018An analytic theory of generalization dynamics and transfer learning in deep linear networkscited by 85
  15. 2019Toward Understanding Catastrophic Forgetting in Continual Learningcited by 68
  16. 2020Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descentcited by 68
  17. 2017Learning to select data for transfer learning with Bayesian Optimizationcited by 64
  18. 2018The committee machine: Computational to statistical gaps in learning a two-layers neural networkcited by 58
  19. 2017An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysiscited by 52
  20. 2018Comparing Dynamics: Deep Neural Networks versus Glassy Systemscited by 42
  21. 2017Gradient Descent Learns One-hidden-layer CNN: Don't be Afraid of Spurious Local Minimacited by 40
  22. 2020Data-Dependence of Plateau Phenomenon in Learning with Neural Network --- Statistical Mechanical Analysiscited by 23
  23. 2021Phase Transitions in Transfer Learning for High-Dimensional Perceptronscited by 22
  24. 2019Mean-field inference methods for neural networkscited by 16
  25. 2019Generalization in multitask deep neural classifiers: a statistical physics approachcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 17; 2023: 15; 2024: 17; 2025: 18; 2026: 18.

88 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  2. 2022How catastrophic can catastrophic forgetting be in linear regression?cited by 81
  3. 2022A Theoretical Study on Solving Continual Learningcited by 80
  4. 2023Theory on Forgetting and Generalization of Continual Learningcited by 72
  5. 2022Continual Learning with Foundation Models: An Empirical Study of Latent Replaycited by 61
  6. 2024Theory on Mixture-of-Experts in Continual Learningcited by 54
  7. 2023The Ideal Continual Learner: An Agent That Never Forgetscited by 35
  8. 2023The Tunnel Effect: Building Data Representations in Deep Neural Networkscited by 34
  9. 2023Continual Learning in Linear Classification on Separable Datacited by 34
  10. 2024A Statistical Theory of Regularization-Based Continual Learningcited by 33
  11. 2022The Effect of Task Ordering in Continual Learningcited by 31
  12. 2024Understanding Forgetting in Continual Learning with Linear Regressioncited by 30
38 more
  1. 2022Analysis of Catastrophic Forgetting for Random Orthogonal Transformation Tasks in the Overparameterized Regimecited by 29
  2. 2023Learnability and Algorithm for Continual Learningcited by 28
  3. 2022How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Compositioncited by 27
  4. 2024The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting -- An Analytical Modelcited by 26
  5. 2024CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Modelscited by 25
  6. 2023Fixed Design Analysis of Regularization-Based Continual Learningcited by 24
  7. 2024Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problemcited by 23
  8. 2022Task-Free Continual Learning via Online Discrepancy Distance Learningcited by 22
  9. 2024Disentangling and Mitigating the Impact of Task Similarity for Continual Learningcited by 19
  10. 2022Maslow's Hammer for Catastrophic Forgetting: Node Re-Use vs Node Activationcited by 16
  11. 2022A Theory for Knowledge Transfer in Continual Learningcited by 16
  12. 2025Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuningcited by 16
  13. 2021Learning Curves for Continual Learning in Neural Networks: Self-Knowledge Transfer and Forgettingcited by 14
  14. 2022Provable and Efficient Continual Representation Learningcited by 14
  15. 2024Order parameters and phase transitions of continual learning in deep neural networkscited by 14
  16. 2024Optimal Protocols for Continual Learning via Statistical Physics and Control Theorycited by 14
  17. 2025Optimal Task Order for Continual Learning of Multiple Taskscited by 14
  18. 2022Modelling continual learning in humans with Hebbian context gating and exponentially decaying task signalscited by 12
  19. 2022Continual Variational Autoencoder Learning via Online Cooperative Memorizationcited by 12
  20. 2024Last Iterate Convergence of Incremental Methods and Applications in Continual Learningcited by 10
  21. 2025Lightspeed Geometric Dataset Distance via Sliced Optimal Transportcited by 10
  22. 2022Learning to Learn: How to Continuously Teach Humans and Machinescited by 9
  23. 2022Statistical mechanics of continual learning: variational principle and mean-field potentialcited by 9
  24. 2023Inductive biases of multi-task learning and finetuning: multiple regimes of feature reusecited by 9
  25. 2024Demystifying Language Model Forgetting with Low-rank Example Associationscited by 8
  26. 2024PASS++: A Dual Bias Reduction Framework for Non-Exemplar Class-Incremental Learningcited by 8
  27. 2024Continual Memorization of Factoids in Language Modelscited by 8
  28. 2023Task Relation-aware Continual User Representation Learningcited by 7
  29. 2023Eight challenges in developing theory of intelligencecited by 6
  30. 2025A Theory of Initialisation's Impact on Specialisationcited by 6
  31. 2023The RL Perceptron: Generalisation Dynamics of Policy Learning in High Dimensionscited by 5
  32. 2023Advancing continual lifelong learning in neural information retrieval: definition, dataset, framework, and empirical evaluationcited by 5
  33. 2025Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspectivecited by 5
  34. 2022A theory of learning with constrained weight-distributioncited by 4
  35. 2022Task Formulation Matters When Learning Continually: A Case Study in Visual Question Answeringcited by 4
  36. 2022Navigating Memory Construction by Global Pseudo-Task Simulation for Continual Learningcited by 4
  37. 2024Interpretability of Language Models via Task Spacescited by 4
  38. 2026Putting a Face to Forgetting: Continual Learning meets Mechanistic Interpretabilitycited by 4

50 most cited of 88 All 88 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Remembering Transformer for Continual Learning
  2. 2026Sparsity, Superposition, and Forgetting: A Mechanistic Study of Representation Retention in Continual Learning
  3. 2026Lost or Hidden? A Concept-Level Forgetting in Supervised Continual Learning
  4. 2026Catastrophic Forgetting as Accessibility Collapse: A Three-Level Framework for Knowledge Persistence in Continual Learning
  5. 2022On Generalizing Beyond Domains in Cross-Domain Continual Learning
  6. 2026To Retain or to Adapt? Generalizing Continual Learning
  7. 2021In Defense of the Learning Without Forgetting for Task Incremental Learning
  8. 2024Dual-Phase Continual Learning: Supervised Adaptation Meets Unsupervised Retention