Training Dynamics of Deep Network Linear Regions
Not on the map. Why?
Abstract (arXiv)
The study of Deep Network (DN) training dynamics has largely focused on the evolution of the loss function, evaluated on or around train and test set data points. In fact, many DN phenomenon were first introduced in literature with that respect, e.g., double descent, grokking. In this study, we look at the training dynamics of the input space partition or linear regions formed by continuous piecewise affine DNs, e.g., networks with (leaky)ReLU nonlinearities. First, we present a novel statistic that encompasses the local complexity (LC) of the DN based on the concentration of linear regions inside arbitrary dimensional neighborhoods around data points. We observe that during training, the LC around data points undergoes a number of phases, starting with a decreasing trend after initialization, followed by an ascent and ending with a final descending trend. Using exact visualization methods, we come across the perplexing observation that during the final LC descent phase of training, linear regions migrate away from training and test samples towards the decision boundary, making the DN input-output nearly linear everywhere else. We also observe that the different LC phases are closely related to the memorization and generalization performance of the DN, especially during grokking.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 32 of its 42 references that are in the map. First posted on arXiv 19 Oct 2023.
Builds on
32 of 42 references.
References · 32
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2016Understanding deep learning requires rethinking generalizationcited by 1,674
- 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
- 2017A Closer Look at Memorization in Deep Networkscited by 942
- 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
- 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
- 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 425
- 2014On the Number of Linear Regions of Deep Neural Networkscited by 381
- 2016On the Expressive Power of Deep Neural Networkscited by 306
- 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
- 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
- 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
20 more
- 2019Complexity of Linear Regions in Deep Networkscited by 167
- 2020Implicit Gradient Regularizationcited by 146
- 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 146
- 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 119
- 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
- 2021Learning in High Dimension Always Amounts to Extrapolationcited by 66
- 2018Data augmentation instead of explicit regularizationcited by 63
- 2019The Geometry of Deep Networks: Power Diagram Subdivisioncited by 53
- 2023The SSL Interplay: Augmentations, Inductive Bias, and Generalizationcited by 40
- 2019Identity Crisis: Memorization and Generalization under Extreme Overparameterizationcited by 33
- 2023SplineCam: Exact Visualization and Characterization of Deep Network Geometry and Decision Boundariescited by 31
- 2021MaGNET: Uniform Sampling from Deep Generative Network Manifolds Without Retrainingcited by 23
- 2022Polarity Sampling: Quality and Diversity Control of Pre-Trained Generative Networks via Singular Valuescited by 21
- 2022A Blessing of Dimensionality in Membership Inference through Regularizationcited by 18
- 2023Towards Democratizing Joint-Embedding Self-Supervised Learningcited by 18
- 2022Singular Value Perturbation and Deep Network Optimizationcited by 11
- 2023FAIR-Ensemble: When Fairness Naturally Emerges From Deep Ensemblingcited by 11
- 2019Implicit Regularization in Over-parameterized Neural Networkscited by 10
- 2022Batch Normalization Explainedcited by 8
- 2021Max-Affine Spline Insights Into Deep Network Pruningcited by 5
Built on by
Citing papers per year: 2024: 1; 2025: 2.
3 papers.