# Luck Matters: Understanding Training Dynamics of Deep ReLU Networks

arXiv: https://arxiv.org/abs/1905.13405 · HTML: https://mlsanity.com/paper/1905.13405/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 May 2019.

## Built on by (most cited first, 8 of 8)

- [The Neural Race Reduction: Dynamics of Abstraction in Gated Networks](https://mlsanity.com/paper/2207.10430/) (2022)
- [Understanding the Role of Nonlinearity in Training Dynamics of Contrastive Learning](https://mlsanity.com/paper/2206.01342/) (2022)
- [CLUES: Collaborative High-Quality Data Selection for LLMs via Training Dynamics](https://mlsanity.com/paper/2507.03004/) (2025)
- [Understanding Robustness in Teacher-Student Setting: A New Perspective](https://mlsanity.com/paper/2102.13170/) (2021)
- [The Topos of Transformer Networks](https://mlsanity.com/paper/2403.18415/) (2024)
- [Directional Convergence Analysis under Spherically Symmetric Distribution](https://mlsanity.com/paper/2105.03879/) (2021)
- [Data Augmentation Techniques to Reverse-Engineer Neural Network Weights from Input-Output Queries](https://mlsanity.com/paper/2511.20312/) (2025)
- [Teacher Geometry Shapes Learnability in Teacher-Student Networks](https://mlsanity.com/paper/2609.09595/) (2026)

## Abstract (arXiv)

We analyze the dynamics of training deep ReLU networks and their implications on generalization capability. Using a teacher-student setting, we discovered a novel relationship between the gradient received by hidden student nodes and the activations of teacher nodes for deep ReLU networks. With this relationship and the assumption of small overlapping teacher node activations, we prove that (1) student nodes whose weights are initialized to be close to teacher nodes converge to them at a faster rate, and (2) in over-parameterized regimes and 2-layer case, while a small set of lucky nodes do converge to the teacher nodes, the fan-out weights of other nodes converge to zero. This framework provides insight into multiple puzzling phenomena in deep learning like over-parameterization, implicit regularization, lottery tickets, etc. We verify our assumption by showing that the majority of BatchNorm biases of pre-trained VGG11/16 models are negative. Experiments on (1) random deep teacher networks with Gaussian inputs, (2) teacher network pre-trained on CIFAR-10 and (3) extensive ablation studies validate our multiple theoretical predictions.