# Tight and Efficient Upper Bound on Spectral Norm of Convolutional Layers

arXiv: https://arxiv.org/abs/2409.11859 · HTML: https://mlsanity.com/paper/2409.11859/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 33 references that are in the map.
- First posted on arXiv 18 Sep 2024.

## Builds on (most cited first, 19 of 19)

- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Lipschitz regularity of deep neural networks: analysis and efficient estimation](https://mlsanity.com/paper/1805.10965/) (2018)
- [A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks](https://mlsanity.com/paper/1707.09564/) (2017)
- [Parseval Networks: Improving Robustness to Adversarial Examples](https://mlsanity.com/paper/1704.08847/) (2017)
- [Plug-and-Play Methods Provably Converge with Properly Trained Denoisers](https://mlsanity.com/paper/1905.05406/) (2019)
- [Spectral Norm Regularization for Improving the Generalizability of Deep Learning](https://mlsanity.com/paper/1705.10941/) (2017)
- [Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks](https://mlsanity.com/paper/1806.05393/) (2018)
- [Orthogonal Convolutional Neural Networks](https://mlsanity.com/paper/1911.12207/) (2019)
- [The Singular Values of Convolutional Layers](https://mlsanity.com/paper/1805.10408/) (2018)
- [Generalizable Adversarial Training via Spectral Normalization](https://mlsanity.com/paper/1811.07457/) (2018)
- [Skew Orthogonal Convolutions](https://mlsanity.com/paper/2105.11417/) (2021)
- [A Dynamical System Perspective for Lipschitz Neural Networks](https://mlsanity.com/paper/2110.12690/) (2021)
- [Improved deterministic l2 robustness on CIFAR-10 and CIFAR-100](https://mlsanity.com/paper/2108.04062/) (2021)
- [Efficient Bound of Lipschitz Constant for Convolutional Layers by Gram Iteration](https://mlsanity.com/paper/2305.16173/) (2023)
- [Towards Practical Control of Singular Values of Convolutional Layers](https://mlsanity.com/paper/2211.13771/) (2022)
- [Spectrum Extraction and Clipping for Implicitly Linear Layers](https://mlsanity.com/paper/2402.16017/) (2024)
- [Spectral Norm of Convolutional Layers with Circular and Zero Paddings](https://mlsanity.com/paper/2402.00240/) (2024)
- [Best rank-k approximations for tensors: generalizing Eckart-Young](https://mlsanity.com/paper/1711.06443/) (2017)
- [Spectral norm of a symmetric tensor and its computation](https://mlsanity.com/paper/1808.03864/) (2018)

## Built on by (most cited first, 7 of 7)

- [An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architectures](https://mlsanity.com/paper/2501.07930/) (2025)
- [Improved Scalable Lipschitz Bounds for Deep Neural Networks](https://mlsanity.com/paper/2503.14297/) (2025)
- [On the Interaction of Compressibility and Adversarial Robustness](https://mlsanity.com/paper/2507.17725/) (2025)
- [Unpaired Image-to-Image Translation for Segmentation and Signal Unmixing](https://mlsanity.com/paper/2505.20746/) (2025)
- [Tensorion: A Tensor-Aware Generalization of the Muon Optimizer](https://mlsanity.com/paper/2606.25975/) (2026)
- [Orthogonium : A Unified, Efficient Library of Orthogonal and 1-Lipschitz Building Blocks](https://mlsanity.com/paper/2601.13776/) (2026)
- [Lipschitz-Based Robustness Certification Under Floating-Point Execution](https://mlsanity.com/paper/2603.13334/) (2026)

## Abstract (arXiv)

Controlling the spectral norm of the Jacobian matrix, which is related to the convolution operation, has been shown to improve generalization, training stability and robustness in CNNs. Existing methods for computing the norm either tend to overestimate it or their performance may deteriorate quickly with increasing the input and kernel sizes. In this paper, we demonstrate that the tensor version of the spectral norm of a four-dimensional convolution kernel, up to a constant factor, serves as an upper bound for the spectral norm of the Jacobian matrix associated with the convolution operation. This new upper bound is independent of the input image resolution, differentiable and can be efficiently calculated during training. Through experiments, we demonstrate how this new bound can be used to improve the performance of convolutional architectures.