# Critical Learning Periods Emerge Even in Deep Linear Networks

arXiv: https://arxiv.org/abs/2308.12221 · HTML: https://mlsanity.com/paper/2308.12221/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 19 references that are in the map.
- First posted on arXiv 23 Aug 2023.

## Builds on (most cited first, 8 of 8)

- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Implicit Regularization in Deep Matrix Factorization](https://mlsanity.com/paper/1905.13655/) (2019)
- [Implicit Regularization in Matrix Factorization](https://mlsanity.com/paper/1705.09280/) (2017)
- [On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization](https://mlsanity.com/paper/1802.06509/) (2018)
- [A mathematical theory of semantic development in deep neural networks](https://mlsanity.com/paper/1810.10531/) (2018)
- [Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence](https://mlsanity.com/paper/1905.13277/) (2019)
- [The Neural Race Reduction: Dynamics of Abstraction in Gated Networks](https://mlsanity.com/paper/2207.10430/) (2022)
- [Critical Learning Periods for Multisensory Integration in Deep Networks](https://mlsanity.com/paper/2210.04643/) (2022)

## Built on by (most cited first, 13 of 13)

- [Plastic Learning with Deep Fourier Features](https://mlsanity.com/paper/2410.20634/) (2024)
- [DASH: Warm-Starting Neural Network Training in Stationary Settings without Loss of Plasticity](https://mlsanity.com/paper/2410.23495/) (2024)
- [Early Period of Training Impacts Adaptation for Out-of-Distribution Generalization: An Empirical Study](https://mlsanity.com/paper/2403.15210/) (2024)
- [Carpe Diem: Critical Learning Period-Aware Contract-Based Incentives for Federated Learning](https://mlsanity.com/paper/2503.07869/) (2025)
- [One Period to Rule Them All: Identifying Critical Learning Periods in Deep Networks](https://mlsanity.com/paper/2506.15954/) (2025)
- [Critical Windows of Complexity Control: When Transformers Decide to Reason or Memorize](https://mlsanity.com/paper/2605.04396/) (2026)
- [Temporal Regularization Training: Unleashing the Potential of Spiking Neural Networks](https://mlsanity.com/paper/2506.19256/) (2025)
- [The Lock-In Phase Hypothesis: Identity Consolidation as a Precursor to AGI](https://mlsanity.com/paper/2510.20190/) (2025)
- [Winning the Lottery by Preserving Network Training Dynamics with Concrete Ticket Search](https://mlsanity.com/paper/2512.07142/) (2025)
- [Noise-aware Client Selection for carbon-efficient Federated Learning via Gradient Norm Thresholding](https://mlsanity.com/paper/2603.04194/) (2026)
- [Features have life history. And we should care](https://mlsanity.com/paper/2605.18789/) (2026)
- [Spectral Saliency for Machine Unlearning](https://mlsanity.com/paper/2608.15548/) (2026)
- [The Geometry of Refusal: Why Post-Hoc Safety Is Fragile and Pretraining-Time Safety Persists](https://mlsanity.com/paper/2609.06934/) (2026)

## Abstract (arXiv)

Critical learning periods are periods early in development where temporary sensory deficits can have a permanent effect on behavior and learned representations. Despite the radical differences between biological and artificial networks, critical learning periods have been empirically observed in both systems. This suggests that critical periods may be fundamental to learning and not an accident of biology. Yet, why exactly critical periods emerge in deep networks is still an open question, and in particular it is unclear whether the critical periods observed in both systems depend on particular architectural or optimization details. To isolate the key underlying factors, we focus on deep linear network models, and show that, surprisingly, such networks also display much of the behavior seen in biology and artificial networks, while being amenable to analytical treatment. We show that critical periods depend on the depth of the model and structure of the data distribution. We also show analytically and in simulations that the learning of features is tied to competition between sources. Finally, we extend our analysis to multi-task learning to show that pre-training on certain tasks can damage the transfer performance on new tasks, and show how this depends on the relationship between tasks and the duration of the pre-training stage. To the best of our knowledge, our work provides the first analytically tractable model that sheds light into why critical learning periods emerge in biological and artificial networks.