# Capacity Control of ReLU Neural Networks by Basis-path Norm

arXiv: https://arxiv.org/abs/1809.07122 · HTML: https://mlsanity.com/paper/1809.07122/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Sep 2018.

## Built on by (most cited first, 10 of 10)

- [Model Complexity of Deep Learning: A Survey](https://mlsanity.com/paper/2103.05127/) (2021)
- [Small Transformers Compute Universal Metric Embeddings](https://mlsanity.com/paper/2209.06788/) (2022)
- [A path-norm toolkit for modern networks: consequences, promises and challenges](https://mlsanity.com/paper/2310.01225/) (2023)
- [LocalDrop: A Hybrid Regularization for Deep Neural Networks](https://mlsanity.com/paper/2103.00719/) (2021)
- [Accelerating Training of Batch Normalization: A Manifold Perspective](https://mlsanity.com/paper/2101.02916/) (2021)
- [Analytic function approximation by path norm regularized deep networks](https://mlsanity.com/paper/2104.02095/) (2021)
- [Quantifying the Impact of Label Noise on Federated Learning](https://mlsanity.com/paper/2211.07816/) (2022)
- [Hidden Synergy: L1 Weight Normalization and 1-Path-Norm Regularization](https://mlsanity.com/paper/2404.19112/) (2024)
- [A priori Estimates for Deep Residual Network in Continuous-time Reinforcement Learning](https://mlsanity.com/paper/2402.16899/) (2024)
- [Path Regularization: A Near-Complete and Optimal Nonasymptotic Generalization Theory for Multilayer Neural Networks and Double Descent Phenomenon](https://mlsanity.com/paper/2503.02129/) (2025)

## Abstract (arXiv)

Recently, path norm was proposed as a new capacity measure for neural networks with Rectified Linear Unit (ReLU) activation function, which takes the rescaling-invariant property of ReLU into account. It has been shown that the generalization error bound in terms of the path norm explains the empirical generalization behaviors of the ReLU neural networks better than that of other capacity measures. Moreover, optimization algorithms which take path norm as the regularization term to the loss function, like Path-SGD, have been shown to achieve better generalization performance. However, the path norm counts the values of all paths, and hence the capacity measure based on path norm could be improperly influenced by the dependency among different paths. It is also known that each path of a ReLU network can be represented by a small group of linearly independent basis paths with multiplication and division operation, which indicates that the generalization behavior of the network only depends on only a few basis paths. Motivated by this, we propose a new norm Basis-path Norm based on a group of linearly independent paths to measure the capacity of neural networks more accurately. We establish a generalization error bound based on this basis path norm, and show it explains the generalization behaviors of ReLU networks more accurately than previous capacity measures via extensive experiments. In addition, we develop optimization algorithms which minimize the empirical risk regularized by the basis-path norm. Our experiments on benchmark datasets demonstrate that the proposed regularization method achieves clearly better performance on the test set than the previous regularization approaches.