mlsanityData to 13 Oct 2026

Spectral Norm of Convolutional Layers with Circular and Zero Paddings

arXiv 2402.00240 ↗First posted 31 Jan 2024cs.LG cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

This paper leverages the use of Gram iteration an efficient, deterministic, and differentiable method for computing spectral norm with an upper bound guarantee. Designed for circular convolutional layers, we generalize the use of the Gram iteration to zero padding convolutional layers and prove its quadratic convergence. We also provide theorems for bridging the gap between circular and zero padding convolution's spectral norm. We design a spectral rescaling that can be used as a competitive 1-Lipschitz layer that enhances network robustness. Demonstrated through experiments, our method outperforms state-of-the-art techniques in precision, computational cost, and scalability. The code of experiments is available at https://github.com/blaisedelattre/lip4conv.

Key facts. Cited by 4 papers in the map (data through 13 Oct 2026). Builds on 33 of its 46 references that are in the map. First posted on arXiv 31 Jan 2024.

Builds on

33 of 46 references.

References · 33

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,870
  2. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,440
  3. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,621
  4. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,130
  5. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  6. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 413
  7. 2020A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospectscited by 396
  8. 2019Making Convolutional Networks Shift-Invariant Againcited by 391
  9. 2019Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networkscited by 292
  10. 2017Parseval Networks: Improving Robustness to Adversarial Examplescited by 291
  11. 2019Plug-and-Play Methods Provably Converge with Properly Trained Denoiserscited by 197
  12. 2018Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networkscited by 194
21 more
  1. 2020How Much Position Information Do Convolutional Neural Networks Encode?cited by 171
  2. 2020On Translation Invariance in CNNs: Convolutional Layers can Exploit Absolute Spatial Locationcited by 141
  3. 2019Orthogonal Convolutional Neural Networkscited by 123
  4. 2021Orthogonalizing Convolutional Layers with the Cayley Transformcited by 105
  5. 2018The Singular Values of Convolutional Layerscited by 95
  6. 2019Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networkscited by 86
  7. 2016Understanding Deep Convolutional Networkscited by 76
  8. 2018Generalizable Adversarial Training via Spectral Normalizationcited by 68
  9. 2021Skew Orthogonal Convolutionscited by 67
  10. 2021A Dynamical System Perspective for Lipschitz Neural Networkscited by 59
  11. 2023A Unified Algebraic Perspective on Lipschitz Neural Networkscited by 58
  12. 2022Almost-Orthogonal Layers for Efficient General-Purpose Lipschitz Networkscited by 39
  13. 2021Why Approximate Matrix Square Root Outperforms Accurate SVD in Global Covariance Pooling?cited by 24
  14. 2021Robust Differentiable SVDcited by 19
  15. 2023Efficient Bound of Lipschitz Constant for Convolutional Layers by Gram Iterationcited by 19
  16. 2021Existence, Stability and Scalability of Orthogonal Convolutional Neural Networkscited by 16
  17. 2022Defects of Convolutional Decoder Networks in Frequency Representationcited by 10
  18. 2023The Hessian perspective into the Nature of Convolutional Neural Networkscited by 10
  19. 2023Linear CNNs Discover the Statistical Structure of the Dataset Using Only the Most Dominant Frequenciescited by 7
  20. 2018Bounding Multivariate Trigonometric Polynomials with Applications to Filter Bank Designcited by 5
  21. 2023On the Interplay of Convolutional Padding and Adversarial Robustnesscited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 2.

4 papers.

  1. 2025An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architecturescited by 8
  2. 2024Tight and Efficient Upper Bound on Spectral Norm of Convolutional Layerscited by 7
  3. 2026Orthogonium : A Unified, Efficient Library of Orthogonal and 1-Lipschitz Building Blockscited by 0
  4. 2026Lipschitz-Based Robustness Certification Under Floating-Point Executioncited by 0