mlsanityData to 14 Oct 2026

A path-norm toolkit for modern networks: consequences, promises and challenges

arXiv 2310.01225 ↗First posted 2 Oct 2023stat.ML cs.LG math.STcited by 10

Not on the map. Why?

Abstract (arXiv)

This work introduces the first toolkit around path-norms that fully encompasses general DAG ReLU networks with biases, skip connections and any operation based on the extraction of order statistics: max pooling, GroupSort etc. This toolkit notably allows us to establish generalization bounds for modern neural networks that are not only the most widely applicable path-norm based ones, but also recover or beat the sharpest known bounds of this type. These extended path-norms further enjoy the usual benefits of path-norms: ease of computation, invariance under the symmetries of the network, and improved sharpness on layered fully-connected networks compared to the product of operator norms, another complexity measure most commonly used. The versatility of the toolkit and its ease of implementation allow us to challenge the concrete promises of path-norm-based generalization bounds, by numerically evaluating the sharpest known bounds for ResNets on ImageNet.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 30 of its 53 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

30 of 53 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  3. 2017Exploring Generalization in Deep Learningcited by 600
  4. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
  5. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  6. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 331
  7. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 314
  8. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  9. 2017Size-Independent Sample Complexity of Neural Networkscited by 290
  10. 2016Understanding Deep Neural Networks with Rectified Linear Unitscited by 257
  11. 2021User-friendly introduction to PAC-Bayes boundscited by 239
  12. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 224
18 more
  1. 2018Sorting out Lipschitz function approximationcited by 213
  2. 2020Pruning Neural Networks at Initialization: Why are We Missing the Mark?cited by 182
  3. 2017Generalization in Deep Learningcited by 146
  4. 2016A vector-contraction inequality for Rademacher complexitiescited by 142
  5. 2019A Primer on PAC-Bayesian Learningcited by 132
  6. 2020The Early Phase of Neural Network Trainingcited by 115
  7. 2017Implicit Regularization in Deep Learningcited by 90
  8. 2020In Search of Robust Measures of Generalizationcited by 79
  9. 2019Lipschitz Certificates for Layered Network Structures Driven by Averaged Activation Operatorscited by 55
  10. 2023Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flowscited by 45
  11. 2020Generalization bounds for deep learningcited by 38
  12. 2023Fast, Differentiable and Sparse Top-k: a Convex Analysis Perspectivecited by 26
  13. 2021An Embedding of ReLU Networks and an Analysis of their Identifiabilitycited by 20
  14. 2019Complexity, Statistical Risk, and Metric Entropy of Deep Nets Using Total Path Variationcited by 16
  15. 2022Local Identifiability of Deep ReLU Neural Networks: the Theorycited by 12
  16. 2018Capacity Control of ReLU Neural Networks by Basis-path Normcited by 10
  17. 2022Approximation speed of quantized vs. unquantized ReLU neural networks and beyondcited by 8
  18. 2023Norm-based Generalization Bounds for Compositionally Sparse Neural Networkscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 2.

10 papers.

  1. 2024Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problemscited by 10
  2. 2024How DNNs break the Curse of Dimensionality: Compositionality and Symmetry Learningcited by 6
  3. 2025Convexity in ReLU Neural Networks: beyond ICNNs?cited by 6
  4. 2026Path-conditioned training: a principled way to rescale ReLU neural networkscited by 6
  5. 2024A Rescaling-Invariant Lipschitz Bound Based on Path-Metrics for Modern ReLU Network Parameterizationscited by 4
  6. 2025Intrinsic training dynamics of deep neural networkscited by 4
  7. 2025Non-Vacuous Generalization Bounds: Can Rescaling Invariances Help?cited by 1
  8. 2025Path Regularization: A Near-Complete and Optimal Nonasymptotic Generalization Theory for Multilayer Neural Networks and Double Descent Phenomenoncited by 0
  9. 2025Position: Many generalization measures for deep learning are fragilecited by 0
  10. 2026Rank and computation of the pathlifting Jacobian of a DAG ReLU networkcited by 0