mlsanityData to 7 Oct 2026

Generalization in Deep Networks: The Role of Distance from Initialization

arXiv 1901.01672 ↗First posted 7 Jan 2019cs.LG cs.AI stat.MLcited by 49

Not on the map. Why?

Abstract (arXiv)

Why does training deep neural networks using stochastic gradient descent (SGD) result in a generalization error that does not worsen with the number of parameters in the network? To answer this question, we advocate a notion of effective model capacity that is dependent on em a given random initialization of the network and not just the training algorithm and the data distribution. We provide empirical evidences that demonstrate that the model capacity of SGD-trained deep networks is in fact restricted through implicit regularization of em the ℓ2 distance from the initialization. We also provide theoretical arguments that further highlight the need for initialization-dependent notions of model capacity. We leave as open questions how and why distance from initialization is regularized, and whether it is sufficient to explain generalization.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 14; 2022: 18; 2023: 5; 2024: 7; 2025: 3; 2026: 1.

49 papers.

Sort:

By area:

  1. 2021Physics-informed neural networks with hard constraints for inverse designcited by 277
  2. 2021Assessing Generalization of SGD via Disagreementcited by 116
  3. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  4. 2022Why neural networks find simple solutions: the many regularizers of geometric complexitycited by 38
  5. 2022Robust Fine-Tuning of Deep Neural Networks with Hessian-based Generalization Guaranteescited by 34
  6. 2022Sparse Double Descent: Where Network Pruning Aggravates Overfittingcited by 26
  7. 2021Measuring Generalization with Optimal Transportcited by 24
  8. 2024What Do Learning Dynamics Reveal About Generalization in LLM Reasoning?cited by 22
  9. 2023Zero-Shot Neural Architecture Search: Challenges, Solutions, and Opportunitiescited by 21
  10. 2023Fantastic Generalization Measures are Nowhere to be Foundcited by 19
  11. 2021A Theoretical Analysis of Fine-tuning with Linear Teacherscited by 17
  12. 2022The activity-weight duality in feed forward neural networks: The geometric determinants of generalizationcited by 12
37 more
  1. 2024On the Diminishing Returns of Width for Continual Learningcited by 12
  2. 2022Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal Statescited by 11
  3. 2021Robustness to Pruning Predicts Generalization in Deep Neural Networkscited by 10
  4. 2023Understanding prompt engineering may not require rethinking generalizationcited by 10
  5. 2021Learning an Explicit Hyperparameter Prediction Function Conditioned on Taskscited by 9
  6. 2022On Measuring Excess Capacity in Neural Networkscited by 9
  7. 2022Predicting the generalization gap in neural networks using topological data analysiscited by 9
  8. 2022Analyzing Lottery Ticket Hypothesis from PAC-Bayesian Theory Perspectivecited by 9
  9. 2022Self-Distillation for Further Pre-training of Transformerscited by 9
  10. 2022Task Discovery: Finding the Tasks that Neural Networks Generalize oncited by 9
  11. 2021Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networkscited by 7
  12. 2022Weight Expansion: A New Perspective on Dropout and Generalizationcited by 6
  13. 2023The Pursuit of Human Labeling: A New Perspective on Unsupervised Learningcited by 6
  14. 2024What Does Softmax Probability Tell Us about Classifiers Ranking Across Diverse Test Conditions?cited by 6
  15. 2022Investigating Generalization by Controlling Normalized Margincited by 5
  16. 2022Generalization Bounds for Gradient Methods via Discrete and Continuous Priorcited by 5
  17. 2024How many samples are needed to train a deep neural network?cited by 5
  18. 2022Predicting Out-of-Domain Generalization with Neighborhood Invariancecited by 4
  19. 2022On the Sample Complexity of Two-Layer Networks: Lipschitz vs. Element-Wise Lipschitz Activationcited by 4
  20. 2022Accelerated Riemannian Optimization: Handling Constraints with a Prox to Bound Geometric Penaltiescited by 4
  21. 2022Intersection of Parallels as an Early Stopping Criterioncited by 3
  22. 2021AdjointNet: Constraining machine learning models with physics-based codescited by 2
  23. 2021Exploring the Common Principal Subspace of Deep Features in Neural Networkscited by 2
  24. 2021In Search of Probeable Generalization Measurescited by 2
  25. 2025Just One Layer Norm Guarantees Stable Extrapolationcited by 2
  26. 2021Learning and Generalization in Overparameterized Normalizing Flowscited by 1
  27. 2021Practical Assessment of Generalization Performance Robustness for Deep Networks via Contrastive Examplescited by 1
  28. 2023Stable Distillation: Regularizing Continued Pre-training for Low-Resource Automatic Speech Recognitioncited by 1
  29. 2025R'enyi Sharpness: A Novel Sharpness that Strongly Correlates with Generalizationcited by 1
  30. 2020Why Adversarial Interaction Creates Non-Homogeneous Patterns: A Pseudo-Reaction-Diffusion Model for Turing Instabilitycited by 0
  31. 2021Training Algorithm Matters for the Performance of Neural Network Potential: A Case Study of Adam and the Kalman Filter Optimizerscited by 0
  32. 2021Cascaded Compressed Sensing Networks: A Reversible Architecture for Layerwise Learningcited by 0
  33. 2024LNPT: Label-free Network Pruning and Trainingcited by 0
  34. 2024Generalization Measures for Zero-Shot Cross-Lingual Transfercited by 0
  35. 2024Is network fragmentation a useful complexity measure?cited by 0
  36. 2025Path Regularization: A Near-Complete and Optimal Nonasymptotic Generalization Theory for Multilayer Neural Networks and Double Descent Phenomenoncited by 0
  37. 2026Towards Initialization-dependent and Non-vacuous Generalization Bounds for Overparameterized Shallow Neural Networkscited by 0