# A Rescaling-Invariant Lipschitz Bound Based on Path-Metrics for Modern ReLU Network Parameterizations

arXiv: https://arxiv.org/abs/2405.15006 · HTML: https://mlsanity.com/paper/2405.15006/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 43 references that are in the map.
- First posted on arXiv 23 May 2024.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Spectrally-normalized margin bounds for neural networks](https://mlsanity.com/paper/1706.08498/) (2017)
- [A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks](https://mlsanity.com/paper/1707.09564/) (2017)
- [Stronger generalization bounds for deep nets via a compression approach](https://mlsanity.com/paper/1802.05296/) (2018)
- [Norm-Based Capacity Control in Neural Networks](https://mlsanity.com/paper/1503.00036/) (2015)
- [Generalization in Deep Learning](https://mlsanity.com/paper/1710.05468/) (2017)
- [The Early Phase of Neural Network Training](https://mlsanity.com/paper/2002.10365/) (2020)
- [Provable Filter Pruning for Efficient Neural Networks](https://mlsanity.com/paper/1911.07412/) (2019)
- [A Modern Look at the Relationship between Sharpness and Generalization](https://mlsanity.com/paper/2302.07011/) (2023)
- [Normalized Flat Minima: Exploring Scale Invariant Definition of Flat Minima for Neural Networks using PAC-Bayesian Analysis](https://mlsanity.com/paper/1901.04653/) (2019)
- [Analysis of the Generalization Error: Empirical Risk Minimization over Deep Artificial Neural Networks Overcomes the Curse of Dimensionality in the Numerical Approximation of Black-Scholes Partial Differential Equations](https://mlsanity.com/paper/1809.03062/) (2018)
- [Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flows](https://mlsanity.com/paper/2307.00144/) (2023)
- [The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networks](https://mlsanity.com/paper/2203.04466/) (2022)
- [Fast as CHITA: Neural Network Pruning with Combinatorial Optimization](https://mlsanity.com/paper/2302.14623/) (2023)
- [Post-training Quantization for Neural Networks with Provable Guarantees](https://mlsanity.com/paper/2201.11113/) (2022)
- [A Greedy Algorithm for Quantizing Neural Networks](https://mlsanity.com/paper/2010.15979/) (2020)
- [An Embedding of ReLU Networks and an Analysis of their Identifiability](https://mlsanity.com/paper/2107.09370/) (2021)
- [Complexity, Statistical Risk, and Metric Entropy of Deep Nets Using Total Path Variation](https://mlsanity.com/paper/1902.00800/) (2019)
- [Local Identifiability of Deep ReLU Neural Networks: the Theory](https://mlsanity.com/paper/2206.07424/) (2022)
- [Generalization Error Bounds for Iterative Recovery Algorithms Unfolded as Neural Networks](https://mlsanity.com/paper/2112.04364/) (2021)
- [A path-norm toolkit for modern networks: consequences, promises and challenges](https://mlsanity.com/paper/2310.01225/) (2023)
- [Approximation speed of quantized vs. unquantized ReLU neural networks and beyond](https://mlsanity.com/paper/2205.11874/) (2022)

## Built on by (most cited first, 4 of 4)

- [Non-Vacuous Generalization Bounds: Can Rescaling Invariances Help?](https://mlsanity.com/paper/2509.26149/) (2025)
- [Similarity search generalisation in contrastive learning with InfoNCE loss](https://mlsanity.com/paper/2607.09405/) (2026)
- [Rank and computation of the pathlifting Jacobian of a DAG ReLU network](https://mlsanity.com/paper/2609.18682/) (2026)
- [Sample-Weighted End-to-End Trace-Norm Geometry for Multitask Learning](https://mlsanity.com/paper/2609.29520/) (2026)

## Abstract (arXiv)

Robustness with respect to weight perturbations underpins guarantees for generalization, pruning and quantization. Existing guarantees rely on Lipschitz bounds in parameter space, cover only plain feed-forward MLPs, and break under the ubiquitous neuron-wise rescaling symmetry of ReLU networks. We prove a new Lipschitz inequality expressed through the ℓ1-path-metric of the weights. The bound is (i) rescaling-invariant by construction and (ii) applies to any ReLU-DAG architecture with any combination of convolutions, skip connections, pooling, and frozen (inference-time) batch-normalization -- thus encompassing ResNets, U-Nets, VGG-style CNNs, and more. By respecting the network's natural symmetries, the new bound strictly sharpens prior parameter-space bounds and can be computed in two forward passes. To illustrate its utility, we derive from it a symmetry-aware pruning criterion and show -- through a proof-of-concept experiment on a ResNet-18 trained on ImageNet -- that its pruning performance matches that of classical magnitude pruning, while becoming totally immune to arbitrary neuron-wise rescalings.