mlsanityData to 7 Oct 2026

A Theory of Initialisation's Impact on Specialisation

arXiv 2503.02526 ↗First posted 4 Mar 2025cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Prior work has demonstrated a consistent tendency in neural networks engaged in continual learning tasks, wherein intermediate task similarity results in the highest levels of catastrophic interference. This phenomenon is attributed to the network's tendency to reuse learned features across tasks. However, this explanation heavily relies on the premise that neuron specialisation occurs, i.e. the emergence of localised representations. Our investigation challenges the validity of this assumption. Using theoretical frameworks for the analysis of neural networks, we show a strong dependence of specialisation on the initial condition. More precisely, we show that weight imbalance and high weight entropy can favour specialised solutions. We then apply these insights in the context of continual learning, first showing the emergence of a monotonic relation between task-similarity and forgetting in non-specialised networks. {Finally, we show that specialization by weight imbalance is beneficial on the commonly employed elastic weight consolidation regularisation technique.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 48 of its 92 references that are in the map. First posted on arXiv 4 Mar 2025.

Builds on

48 of 92 references.

References · 48

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2016Learning without Forgettingcited by 2,465
  5. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  6. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  7. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  8. 2014How transferable are features in deep neural networks?cited by 1,216
  9. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 929
  10. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  11. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  12. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
36 more
  1. 2018On Lazy Training in Differentiable Programmingcited by 671
  2. 2018Disentangling by Factorisingcited by 614
  3. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  4. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  5. 2015Neural Module Networkscited by 375
  6. 2018Understanding disentangling in β-VAEcited by 332
  7. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  8. 2017Variational Inference of Disentangled Latent Concepts from Unlabeled Observationscited by 240
  9. 2020Weakly-Supervised Disentanglement Without Compromisescited by 237
  10. 2019Recurrent Independent Mechanismscited by 219
  11. 2018A mathematical theory of semantic development in deep neural networkscited by 182
  12. 2015Conditional Computation in Neural Networks for faster modelscited by 166
  13. 2020Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semanticscited by 158
  14. 2017Learning to Reason: End-to-End Module Networks for Visual Question Answeringcited by 125
  15. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  16. 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
  17. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
  18. 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 86
  19. 2018An analytic theory of generalization dynamics and transfer learning in deep linear networkscited by 84
  20. 2019Measuring Compositionality in Representation Learningcited by 84
  21. 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 69
  22. 2018Explainable Neural Computation via Stack Neural Module Networkscited by 54
  23. 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 46
  24. 2022Neurocompositional computing: From the Central Paradox of Cognition to a new generation of AI systemscited by 44
  25. 2023Break It Down: Evidence for Structural Compositionality in Neural Networkscited by 44
  26. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  27. 2022The Neural Race Reduction: Dynamics of Abstraction in Gated Networkscited by 37
  28. 20243DRealCar: An In-the-wild RGB-D Car Dataset with 360-degree Viewscited by 37
  29. 2018Automatically Composing Representation Transformations as a Means for Generalizationcited by 35
  30. 2021Learning disentangled representations via product manifold projectioncited by 20
  31. 2022Maslow's Hammer for Catastrophic Forgetting: Node Re-Use vs Node Activationcited by 16
  32. 2024Why Do Animals Need Shaping? A Theory of Task Composition and Curriculum Learningcited by 16
  33. 2024On The Specialization of Neural Modulescited by 14
  34. 2021Distinguishing rule- and exemplar-based generalization in learning systemscited by 12
  35. 2024Bias in Motion: Theoretical Insights into the Dynamics of Bias in SGD Trainingcited by 8
  36. 2019Variational Learning with Disentanglement-PyTorchcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2026A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuningcited by 5
  2. 2025Analytic theory of dropout regularizationcited by 2
  3. 2026Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networkscited by 2
  4. 2026Signal-to-Noise Ratio and Sample Size Govern Representational Alignment in Neural Networkscited by 0
  5. 2026λ-JEPA Spectral Anti-Collapse Regularization for Self-Supervised Learningcited by 0
  6. 2026A Dynamical Theory of LoRA in Continual Learningcited by 0