mlsanityData to 13 Oct 2026

The Singular Values of Convolutional Layers

arXiv 1805.10408 ↗First posted 26 May 2018cs.LG cs.AI stat.MLcited by 95

Not on the map. Why?

Abstract (arXiv)

We characterize the singular values of the linear transformation associated with a standard 2D multi-channel convolutional layer, enabling their efficient computation. This characterization also leads to an algorithm for projecting a convolutional layer onto an operator-norm ball. We show that this is an effective regularizer; for example, it improves the test error of a deep residual network using batch normalization on CIFAR-10 from 6.2% to 5.3%.

Key facts. Cited by 95 papers in the map (data through 13 Oct 2026). First posted on arXiv 26 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 24; 2022: 21; 2023: 16; 2024: 10; 2025: 14; 2026: 10.

95 papers.

Sort:

By area:

  1. 2021Globally-Robust Neural Networkscited by 107
  2. 2021Orthogonalizing Convolutional Layers with the Cayley Transformcited by 105
  3. 2021Skew Orthogonal Convolutionscited by 67
  4. 2021A Dynamical System Perspective for Lipschitz Neural Networkscited by 59
  5. 2023GibbsDDRM: A Partially Collapsed Gibbs Sampler for Solving Blind Inverse Problems with Denoising Diffusion Restorationcited by 59
  6. 2022Singular Value Fine-tuning: Few-shot Segmentation requires Few-parameters Fine-tuningcited by 50
  7. 2021Improved deterministic l2 robustness on CIFAR-10 and CIFAR-100cited by 47
  8. 2022Almost-Orthogonal Layers for Efficient General-Purpose Lipschitz Networkscited by 39
  9. 2022LOT: Layer-wise Orthogonal Training on Improving ℓ2 Certified Robustnesscited by 32
  10. 2023Learning Weakly Convex Regularizers for Convergent Image-Reconstruction Algorithmscited by 30
  11. 2022projUNN: efficient method for training deep networks with unitary matricescited by 27
  12. 2021Reverse Differentiation via Predictive Codingcited by 23
38 more
  1. 2023Efficient Bound of Lipschitz Constant for Convolutional Layers by Gram Iterationcited by 19
  2. 2021Scaling Properties of Deep Residual Networkscited by 16
  3. 2021Existence, Stability and Scalability of Orthogonal Convolutional Neural Networkscited by 16
  4. 2021Convolutional Normalization: Improving Deep Convolutional Network Robustness and Trainingcited by 15
  5. 2021A Probabilistic Approach to Neural Network Pruningcited by 15
  6. 2022Approximation of Lipschitz Functions using Deep Spline Neural Networkscited by 15
  7. 2021Predictive Coding Can Do Exact Backpropagation on Convolutional and Recurrent Neural Networkscited by 14
  8. 2022Neural network training under semidefinite constraintscited by 11
  9. 2023The Hessian perspective into the Nature of Convolutional Neural Networkscited by 10
  10. 2022On Measuring Excess Capacity in Neural Networkscited by 9
  11. 2022Improved techniques for deterministic l2 robustnesscited by 9
  12. 2022Towards Practical Control of Singular Values of Convolutional Layerscited by 9
  13. 2024Analysis of Linear Mode Connectivity via Permutation-Based Weight Matching: With Insights into Other Permutation Search Methodscited by 9
  14. 2022Few-Shot Learning by Dimensionality Reduction in Gradient Spacecited by 8
  15. 2022The SVD of Convolutional Weights: A CNN Interpretability Frameworkcited by 8
  16. 2024Unitary convolutions for learning on graphs and groupscited by 8
  17. 2025An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architecturescited by 8
  18. 2021Scaling-up Diverse Orthogonal Convolutional Networks with a Paraunitary Frameworkcited by 7
  19. 2021Robustness against Adversarial Attacks in Neural Networks using Incremental Dissipativitycited by 7
  20. 2023Linear CNNs Discover the Statistical Structure of the Dataset Using Only the Most Dominant Frequenciescited by 7
  21. 2024Spectrum Extraction and Clipping for Implicitly Linear Layerscited by 7
  22. 2024Tight and Efficient Upper Bound on Spectral Norm of Convolutional Layerscited by 7
  23. 2021Spectral Tensor Train Parameterization of Deep Learning Layerscited by 6
  24. 2021EZCrop: Energy-Zoned Channels for Robust Output Pruningcited by 6
  25. 2021Stable Long-Term Recurrent Video Super-Resolutioncited by 6
  26. 2022Proximal Residual Flows for Bayesian Inverse Problemscited by 5
  27. 2022Orthogonal SVD Covariance Conditioning and Latent Disentanglementcited by 5
  28. 2024Graph Unitary Message Passingcited by 5
  29. 2025Flow Straight and Fast in Hilbert Space: Functional Rectified Flowcited by 5
  30. 2021MP3net: coherent, minute-long music generation from raw audio with a simple convolutional GANcited by 4
  31. 2022Backpropagation Clipping for Deep Learning with Differential Privacycited by 4
  32. 2022Improving Covariance Conditioning of the SVD Meta-layer by Orthogonalitycited by 4
  33. 2022Deep equilibrium models as estimators for continuous latent variablescited by 4
  34. 2023Computing excited states of molecules using normalizing flowscited by 4
  35. 2024Spectral Norm of Convolutional Layers with Circular and Zero Paddingscited by 4
  36. 2026Spectral Manifold Regularization for Stable and Modular Routing in Deep MoE Architecturescited by 4
  37. 2026Towards A Unified PAC-Bayesian Framework for Norm-based Generalization Boundscited by 4
  38. 2021Certified Defense via Latent Space Randomized Smoothing with Orthogonal Encoderscited by 3

50 most cited of 95 All 95 (JSON)