mlsanityData to 7 Oct 2026

Geometry of Optimization and Implicit Regularization in Deep Learning

arXiv 1705.03071 ↗First posted 8 May 2017cs.LGcited by 39

Not on the map. Why?

Abstract (arXiv)

We argue that the optimization plays a crucial role in generalization of deep learning models through implicit regularization. We do this by demonstrating that generalization ability is not controlled by network size but rather by some other implicit control. We then demonstrate how changing the empirical optimization procedure can improve generalization, even if actual optimization quality is not affected. We do so by studying the geometry of the parameter space of deep networks, and devising an optimization algorithm attuned to this geometry.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 May 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 6; 2023: 10; 2024: 3; 2025: 3; 2026: 4.

39 papers.

Sort:

By area:

  1. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  2. 2021What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theorycited by 60
  3. 2021ARAPReg: An As-Rigid-As Possible Regularization Loss for Learning Deformable Shape Generatorscited by 40
  4. 2022How Much Data Are Augmentations Worth? An Investigation into Scaling Laws, Invariance, and Implicit Regularizationcited by 40
  5. 2022Training invariances and the low-rank phenomenon: beyond linear networkscited by 33
  6. 2022Fast Convex Optimization for Two-Layer ReLU Networks: Equivalent Model Classes and Cone Decompositionscited by 31
  7. 2021Implicit Sparse Regularization: The Impact of Depth and Early Stoppingcited by 30
  8. 2021More is Less: Inducing Sparsity via Overparameterizationcited by 22
  9. 2021Reproducing Activation Function for Deep Learningcited by 19
  10. 2023Regularization and Optimal Multiclass Learningcited by 15
  11. 2021The Discovery of Dynamics via Linear Multistep Methods and Deep Learning: Error Estimationcited by 14
  12. 2022Spectral Bias Outside the Training Set for Deep Networks in the Kernel Regimecited by 11
27 more
  1. 2021Deep Learning Generalization and the Convex Hull of Training Setscited by 10
  2. 2021SGD Implicitly Regularizes Generalization Errorcited by 10
  3. 2023Robust Implicit Regularization via Weight Normalizationcited by 10
  4. 2024Pretraining Decision Transformers with Reward Prediction for In-Context Multi-task Structured Bandit Learningcited by 10
  5. 2022Implicit Bias of MSE Gradient Optimization in Underparameterized Neural Networkscited by 9
  6. 2021SGD with a Constant Large Learning Rate Can Converge to Local Maximacited by 8
  7. 2023Implicit Regularization for Group Sparsitycited by 8
  8. 2023PRIOR: Personalized Prior for Reactivating the Information Overlooked in Federated Learningcited by 8
  9. 2021On the Double Descent of Random Features Models Trained with SGDcited by 5
  10. 2021Minimum complexity interpolation in random features modelscited by 4
  11. 2023Implicit regularization in AI meets generalized hardness of approximation in optimization -- Sharp results for diagonal linear networkscited by 4
  12. 2024A Bayesian Model Selection Criterion for Selecting Pretraining Checkpointscited by 4
  13. 2023On the curvature of the loss landscapecited by 3
  14. 2023How Graph Neural Networks Learn: Lessons from Training Dynamicscited by 3
  15. 2023The Effect of SGD Batch Size on Autoencoder Learning: Sparsity, Sharpness, and Feature Learningcited by 2
  16. 2024Depth Separation in Norm-Bounded Infinite-Width Neural Networkscited by 2
  17. 2025Deep Learning Optimization Using Self-Adaptive Weighted Auxiliary Variablescited by 2
  18. 2025Deconstructing Attention: Investigating Design Principles for Effective Language Modelingcited by 2
  19. 2021PACMAN: PAC-style bounds accounting for the Mismatch between Accuracy and Negative log-losscited by 1
  20. 2026Merging Beyond: Streaming LLM Updates via Activation-Guided Rotationscited by 1
  21. 2026Model Merging on Loss Landscape: A Geometry Perspectivecited by 1
  22. 2021Comments on Leo Breiman's paper 'Statistical Modeling: The Two Cultures' (Statistical Science, 2001, 16(3), 199-231)cited by 0
  23. 2022Iterative regularization in classification via hinge loss diagonal descentcited by 0
  24. 2023Convex Dual Theory Analysis of Two-Layer Convolutional Neural Networks with Soft-Thresholdingcited by 0
  25. 2025Optimal Implicit Bias in Linear Regressioncited by 0
  26. 2026Self-Regularized Learning Methodscited by 0
  27. 2026Trust, but Verify: Peeling Low-Bit Transformer Networks for Training Monitoringcited by 0