mlsanityData to 7 Oct 2026

Path-SGD: Path-Normalized Optimization in Deep Neural Networks

arXiv 1506.02617 ↗First posted 8 Jun 2015cs.LG cs.CV cs.NEcited by 117

Not on the map. Why?

Abstract (arXiv)

We revisit the choice of SGD for training deep neural networks by reconsidering the appropriate geometry in which to optimize the weights. We argue for a geometry invariant to rescaling of weights that does not affect the output of the network, and suggest Path-SGD, which is an approximate steepest descent method with respect to a path-wise regularizer related to max-norm regularization. Path-SGD is easy and efficient to implement and leads to empirical gains over SGD and AdaGrad.

Key facts. Cited by 117 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Jun 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 19; 2022: 21; 2023: 18; 2024: 26; 2025: 11; 2026: 22.

117 papers.

Sort:

By area:

  1. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  2. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  3. 2023Equivariant Architectures for Learning in Deep Weight Spacescited by 106
  4. 2022Vision Transformers provably learn spatial structurecited by 92
  5. 2021GBK-GNN: Gated Bi-Kernel Graph Neural Networks for Modeling Both Homophily and Heterophilycited by 84
  6. 2024Scalable Optimization in the Modular Normcited by 64
  7. 2021What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theorycited by 60
  8. 2021Understanding neural networks with reproducing kernel Banach spacescited by 50
  9. 2023Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networkscited by 50
  10. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 47
  11. 2024Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitatecited by 46
  12. 2023Solving Regularized Exp, Cosh and Sinh Regression Problemscited by 44
38 more
  1. 2021Near-Minimax Optimal Estimation With Shallow ReLU Neural Networkscited by 40
  2. 2022Towards understanding how momentum improves generalization in deep learningcited by 40
  3. 2022Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscapecited by 39
  4. 2023Hidden symmetries of ReLU networkscited by 38
  5. 2022Symmetries, flat minima, and the conserved quantities of gradient flowcited by 35
  6. 2022Training invariances and the low-rank phenomenon: beyond linear networkscited by 33
  7. 2025Symmetry in Neural Network Parameter Spacescited by 31
  8. 2025Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?cited by 30
  9. 2022Symmetry Teleportation for Accelerated Optimizationcited by 25
  10. 2023Nonparametric regression using over-parameterized shallow ReLU neural networkscited by 24
  11. 2023Improving Convergence and Generalization Using Parameter Symmetriescited by 23
  12. 2024Scale Equivariant Graph Metanetworkscited by 23
  13. 2022Approximation bounds for norm constrained neural networks with applications to regression and GANscited by 21
  14. 2024The Empirical Impact of Neural Parameter Symmetries, or Lack Thereofcited by 21
  15. 2021An Embedding of ReLU Networks and an Analysis of their Identifiabilitycited by 20
  16. 2023Rethinking White-Box Watermarks on Deep Learning Models under Neural Structural Obfuscationcited by 20
  17. 2023Fantastic Generalization Measures are Nowhere to be Foundcited by 19
  18. 2023The Geometry of Neural Nets' Parameter Spaces Under Reparametrizationcited by 16
  19. 2025Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusioncited by 16
  20. 2024Monomial Matrix Group Equivariant Neural Functional Networkscited by 15
  21. 2021Path Regularization: A Convexity and Sparsity Inducing Regularization for Parallel ReLU Networkscited by 14
  22. 2023Quantum Machine Learning on Near-Term Quantum Devices: Current State of Supervised and Unsupervised Techniques for Real-World Applicationscited by 14
  23. 2022Instance-Dependent Generalization Bounds via Optimal Transportcited by 13
  24. 2024Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networkscited by 13
  25. 2022Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Surveycited by 12
  26. 2022Local Identifiability of Deep ReLU Neural Networks: the Theorycited by 12
  27. 2022Small Transformers Compute Universal Metric Embeddingscited by 12
  28. 2023A Symmetry-Aware Exploration of Bayesian Neural Network Posteriorscited by 12
  29. 2021SGD Implicitly Regularizes Generalization Errorcited by 10
  30. 2024Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problemscited by 9
  31. 2024Natural gradient and parameter estimation for quantum Boltzmann machinescited by 9
  32. 2021Universal approximation and model compression for radial neural networkscited by 7
  33. 2022Scale-invariant Bayesian Neural Networks with Connectivity Tangent Kernelcited by 7
  34. 2023Weighted variation spaces and approximation by shallow ReLU networkscited by 7
  35. 2024The Surprising Harmfulness of Benign Overfitting for Adversarial Robustnesscited by 7
  36. 2024Leveraging PAC-Bayes Theory and Gibbs Distributions for Generalization Bounds with Complexity Measurescited by 7
  37. 2024Boosting Adversarial Training via Fisher-Rao Norm-based Regularizationcited by 7
  38. 2024On the Benefits of Over-parameterization for Out-of-Distribution Generalizationcited by 7

50 most cited of 117 All 117 (JSON)