# Singular Values for ReLU Layers

arXiv: https://arxiv.org/abs/1812.02566 · HTML: https://mlsanity.com/paper/1812.02566/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Dec 2018.

## Built on by (most cited first, 10 of 10)

- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [Approaching Deep Learning through the Spectral Dynamics of Weights](https://mlsanity.com/paper/2408.11804/) (2024)
- [Complete the Missing Half: Augmenting Aggregation Filtering with Diversification for Graph Convolutional Neural Networks](https://mlsanity.com/paper/2212.10822/) (2022)
- [Analytical bounds on the local Lipschitz constants of ReLU networks](https://mlsanity.com/paper/2104.14672/) (2021)
- [The SVD of Convolutional Weights: A CNN Interpretability Framework](https://mlsanity.com/paper/2208.06894/) (2022)
- [Low-Rank Learning by Design: the Role of Network Architecture and Activation Linearity in Gradient Rank Collapse](https://mlsanity.com/paper/2402.06751/) (2024)
- [Convex Geometry of ReLU-layers, Injectivity on the Ball and Local Reconstruction](https://mlsanity.com/paper/2307.09672/) (2023)
- [Uncertainty Quantification for Physics-Informed Neural Networks with Extended Fiducial Inference](https://mlsanity.com/paper/2505.19136/) (2025)
- [Linearization of ReLU Activation Function for Neural Network-Embedded Optimization: Optimal Day-Ahead Energy Scheduling](https://mlsanity.com/paper/2310.01758/) (2023)
- [Extended Fiducial Inference for Individual Treatment Effects via Deep Neural Networks](https://mlsanity.com/paper/2505.01995/) (2025)

## Abstract (arXiv)

Despite their prevalence in neural networks we still lack a thorough theoretical characterization of ReLU layers. This paper aims to further our understanding of ReLU layers by studying how the activation function ReLU interacts with the linear component of the layer and what role this interaction plays in the success of the neural network in achieving its intended task. To this end, we introduce two new tools: ReLU singular values of operators and the Gaussian mean width of operators. By presenting on the one hand theoretical justifications, results, and interpretations of these two concepts and on the other hand numerical experiments and results of the ReLU singular values and the Gaussian mean width being applied to trained neural networks, we hope to give a comprehensive, singular-value-centric view of ReLU layers. We find that ReLU singular values and the Gaussian mean width do not only enable theoretical insights, but also provide one with metrics which seem promising for practical applications. In particular, these measures can be used to distinguish correctly and incorrectly classified data as it traverses the network. We conclude by introducing two tools based on our findings: double-layers and harmonic pruning.