mlsanityData to 7 Oct 2026

Tight and Efficient Upper Bound on Spectral Norm of Convolutional Layers

arXiv 2409.11859 ↗First posted 18 Sep 2024cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Controlling the spectral norm of the Jacobian matrix, which is related to the convolution operation, has been shown to improve generalization, training stability and robustness in CNNs. Existing methods for computing the norm either tend to overestimate it or their performance may deteriorate quickly with increasing the input and kernel sizes. In this paper, we demonstrate that the tensor version of the spectral norm of a four-dimensional convolution kernel, up to a constant factor, serves as an upper bound for the spectral norm of the Jacobian matrix associated with the convolution operation. This new upper bound is independent of the input image resolution, differentiable and can be efficiently calculated during training. Through experiments, we demonstrate how this new bound can be used to improve the performance of convolutional architectures.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 19 of its 33 references that are in the map. First posted on arXiv 18 Sep 2024.

Builds on

19 of 33 references.

References · 19

  1. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  2. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 413
  3. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 309
  4. 2017Parseval Networks: Improving Robustness to Adversarial Examplescited by 288
  5. 2019Plug-and-Play Methods Provably Converge with Properly Trained Denoiserscited by 197
  6. 2017Spectral Norm Regularization for Improving the Generalizability of Deep Learningcited by 178
  7. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  8. 2019Orthogonal Convolutional Neural Networkscited by 122
  9. 2018The Singular Values of Convolutional Layerscited by 95
  10. 2018Generalizable Adversarial Training via Spectral Normalizationcited by 68
  11. 2021Skew Orthogonal Convolutionscited by 67
  12. 2021A Dynamical System Perspective for Lipschitz Neural Networkscited by 59
7 more
  1. 2021Improved deterministic l2 robustness on CIFAR-10 and CIFAR-100cited by 47
  2. 2023Efficient Bound of Lipschitz Constant for Convolutional Layers by Gram Iterationcited by 19
  3. 2022Towards Practical Control of Singular Values of Convolutional Layerscited by 9
  4. 2024Spectrum Extraction and Clipping for Implicitly Linear Layerscited by 7
  5. 2024Spectral Norm of Convolutional Layers with Circular and Zero Paddingscited by 4
  6. 2017Best rank-k approximations for tensors: generalizing Eckart-Youngcited by 3
  7. 2018Spectral norm of a symmetric tensor and its computationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 3.

7 papers.

  1. 2025An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architecturescited by 8
  2. 2025Improved Scalable Lipschitz Bounds for Deep Neural Networkscited by 2
  3. 2025On the Interaction of Compressibility and Adversarial Robustnesscited by 2
  4. 2025Unpaired Image-to-Image Translation for Segmentation and Signal Unmixingcited by 1
  5. 2026Tensorion: A Tensor-Aware Generalization of the Muon Optimizercited by 1
  6. 2026Orthogonium : A Unified, Efficient Library of Orthogonal and 1-Lipschitz Building Blockscited by 0
  7. 2026Lipschitz-Based Robustness Certification Under Floating-Point Executioncited by 0