mlsanityData to 14 Oct 2026

Spectrum Extraction and Clipping for Implicitly Linear Layers

arXiv 2402.16017 ↗First posted 25 Feb 2024cs.LG cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

We show the effectiveness of automatic differentiation in efficiently and correctly computing and controlling the spectrum of implicitly linear operators, a rich family of layer types including all standard convolutional and dense layers. We provide the first clipping method which is correct for general convolution layers, and illuminate the representational limitation that caused correctness issues in prior work. We study the effect of the batch normalization layers when concatenated with convolutional layers and show how our clipping method can be applied to their composition. By comparing the accuracy and performance of our algorithms to the state-of-the-art methods, using various experiments, we show they are more precise and efficient and lead to better generalization and adversarial robustness. We provide the code for using our methods at https://github.com/Ali-E/FastClip.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 24 of its 46 references that are in the map. First posted on arXiv 25 Feb 2024.

Builds on

24 of 46 references.

References · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2013Intriguing properties of neural networkscited by 5,155
  5. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  6. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  7. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  8. 2017Deep Layer Aggregationcited by 436
  9. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 415
  10. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 314
  11. 2017Parseval Networks: Improving Robustness to Adversarial Examplescited by 293
  12. 2018Sorting out Lipschitz function approximationcited by 213
12 more
  1. 2018Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approachcited by 182
  2. 2021Orthogonalizing Convolutional Layers with the Cayley Transformcited by 105
  3. 2018The Singular Values of Convolutional Layerscited by 95
  4. 2018Generalizable Adversarial Training via Spectral Normalizationcited by 68
  5. 2021Skew Orthogonal Convolutionscited by 67
  6. 2020Revisiting Batch Normalization for Improving Corruption Robustnesscited by 55
  7. 2021Improved deterministic l2 robustness on CIFAR-10 and CIFAR-100cited by 47
  8. 2022LOT: Layer-wise Orthogonal Training on Improving ℓ2 Certified Robustnesscited by 33
  9. 2019Batch Normalization is a Cause of Adversarial Vulnerabilitycited by 29
  10. 2023Efficient Bound of Lipschitz Constant for Convolutional Layers by Gram Iterationcited by 19
  11. 2021Existence, Stability and Scalability of Orthogonal Convolutional Neural Networkscited by 16
  12. 2022Towards Practical Control of Singular Values of Convolutional Layerscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 2.

7 papers.

  1. 2026Enhancing LLM Training via Spectral Clippingcited by 10
  2. 2025An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architecturescited by 8
  3. 2024Tight and Efficient Upper Bound on Spectral Norm of Convolutional Layerscited by 7
  4. 2025AMUN: Adversarial Machine UNlearningcited by 2
  5. 2024LOTOS: Layer-wise Orthogonalization for Training Robust Ensemblescited by 1
  6. 2025DeepKoopFormer: A Koopman Enhanced Transformer Based Architecture for Time Series Forecastingcited by 0
  7. 2026Orthogonium : A Unified, Efficient Library of Orthogonal and 1-Lipschitz Building Blockscited by 0