# The Singular Values of Convolutional Layers

arXiv: https://arxiv.org/abs/1805.10408 · HTML: https://mlsanity.com/paper/1805.10408/

## Key facts

- Cited by 95 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 May 2018.

## Built on by (most cited first, 30 of 95)

- [Globally-Robust Neural Networks](https://mlsanity.com/paper/2102.08452/) (2021)
- [Orthogonalizing Convolutional Layers with the Cayley Transform](https://mlsanity.com/paper/2104.07167/) (2021)
- [Skew Orthogonal Convolutions](https://mlsanity.com/paper/2105.11417/) (2021)
- [A Dynamical System Perspective for Lipschitz Neural Networks](https://mlsanity.com/paper/2110.12690/) (2021)
- [GibbsDDRM: A Partially Collapsed Gibbs Sampler for Solving Blind Inverse Problems with Denoising Diffusion Restoration](https://mlsanity.com/paper/2301.12686/) (2023)
- [Singular Value Fine-tuning: Few-shot Segmentation requires Few-parameters Fine-tuning](https://mlsanity.com/paper/2206.06122/) (2022)
- [Improved deterministic l2 robustness on CIFAR-10 and CIFAR-100](https://mlsanity.com/paper/2108.04062/) (2021)
- [Almost-Orthogonal Layers for Efficient General-Purpose Lipschitz Networks](https://mlsanity.com/paper/2208.03160/) (2022)
- [LOT: Layer-wise Orthogonal Training on Improving ℓ2 Certified Robustness](https://mlsanity.com/paper/2210.11620/) (2022)
- [Learning Weakly Convex Regularizers for Convergent Image-Reconstruction Algorithms](https://mlsanity.com/paper/2308.10542/) (2023)
- [projUNN: efficient method for training deep networks with unitary matrices](https://mlsanity.com/paper/2203.05483/) (2022)
- [Reverse Differentiation via Predictive Coding](https://mlsanity.com/paper/2103.04689/) (2021)
- [Efficient Bound of Lipschitz Constant for Convolutional Layers by Gram Iteration](https://mlsanity.com/paper/2305.16173/) (2023)
- [Scaling Properties of Deep Residual Networks](https://mlsanity.com/paper/2105.12245/) (2021)
- [Existence, Stability and Scalability of Orthogonal Convolutional Neural Networks](https://mlsanity.com/paper/2108.05623/) (2021)
- [Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training](https://mlsanity.com/paper/2103.00673/) (2021)
- [A Probabilistic Approach to Neural Network Pruning](https://mlsanity.com/paper/2105.10065/) (2021)
- [Approximation of Lipschitz Functions using Deep Spline Neural Networks](https://mlsanity.com/paper/2204.06233/) (2022)
- [Predictive Coding Can Do Exact Backpropagation on Convolutional and Recurrent Neural Networks](https://mlsanity.com/paper/2103.03725/) (2021)
- [Neural network training under semidefinite constraints](https://mlsanity.com/paper/2201.00632/) (2022)
- [The Hessian perspective into the Nature of Convolutional Neural Networks](https://mlsanity.com/paper/2305.09088/) (2023)
- [On Measuring Excess Capacity in Neural Networks](https://mlsanity.com/paper/2202.08070/) (2022)
- [Improved techniques for deterministic l2 robustness](https://mlsanity.com/paper/2211.08453/) (2022)
- [Towards Practical Control of Singular Values of Convolutional Layers](https://mlsanity.com/paper/2211.13771/) (2022)
- [Analysis of Linear Mode Connectivity via Permutation-Based Weight Matching: With Insights into Other Permutation Search Methods](https://mlsanity.com/paper/2402.04051/) (2024)
- [Few-Shot Learning by Dimensionality Reduction in Gradient Space](https://mlsanity.com/paper/2206.03483/) (2022)
- [The SVD of Convolutional Weights: A CNN Interpretability Framework](https://mlsanity.com/paper/2208.06894/) (2022)
- [Unitary convolutions for learning on graphs and groups](https://mlsanity.com/paper/2410.05499/) (2024)
- [An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architectures](https://mlsanity.com/paper/2501.07930/) (2025)
- [Scaling-up Diverse Orthogonal Convolutional Networks with a Paraunitary Framework](https://mlsanity.com/paper/2106.09121/) (2021)

## Abstract (arXiv)

We characterize the singular values of the linear transformation associated with a standard 2D multi-channel convolutional layer, enabling their efficient computation. This characterization also leads to an algorithm for projecting a convolutional layer onto an operator-norm ball. We show that this is an effective regularizer; for example, it improves the test error of a deep residual network using batch normalization on CIFAR-10 from 6.2% to 5.3%.