mlsanityData to 7 Oct 2026

Towards Practical Control of Singular Values of Convolutional Layers

arXiv 2211.13771 ↗First posted 24 Nov 2022cs.LG cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

In general, convolutional neural networks (CNNs) are easy to train, but their essential properties, such as generalization error and adversarial robustness, are hard to control. Recent research demonstrated that singular values of convolutional layers significantly affect such elusive properties and offered several methods for controlling them. Nevertheless, these methods present an intractable computational challenge or resort to coarse approximations. In this paper, we offer a principled approach to alleviating constraints of the prior art at the expense of an insignificant reduction in layer expressivity. Our method is based on the tensor-train decomposition; it retains control over the actual singular values of convolutional mappings while providing structurally sparse and hardware-friendly representation. We demonstrate the improved properties of modern CNNs with our method and analyze its impact on the model performance, calibration, and adversarial robustness. The source code is available at: https://github.com/WhiteTeaDragon/practical_svd_conv

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 31 of its 43 references that are in the map. First posted on arXiv 24 Nov 2022.

Builds on

31 of 43 references.

References · 31

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2022Generative Adversarial Networkscited by 5,603
  5. 2017On Calibration of Modern Neural Networkscited by 4,860
  6. 2016Wide Residual Networkscited by 3,057
  7. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  8. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  9. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  10. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  11. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  12. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 311
19 more
  1. 2017Parseval Networks: Improving Robustness to Adversarial Examplescited by 288
  2. 2015Tensorizing Neural Networkscited by 246
  3. 2017Spectral Norm Regularization for Improving the Generalizability of Deep Learningcited by 178
  4. 2015Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applicationscited by 161
  5. 2015Structured Pruning of Deep Convolutional Neural Networkscited by 148
  6. 2018The Singular Values of Convolutional Layerscited by 95
  7. 2019Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networkscited by 86
  8. 2021Skew Orthogonal Convolutionscited by 67
  9. 2021On the Practicality of Deterministic Epistemic Uncertaintycited by 62
  10. 2016Ultimate tensorization: compressing convolutional and FC layers alikecited by 55
  11. 2018Wide Compression: Tensor Ring Netscited by 50
  12. 2020Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interferencecited by 47
  13. 2019Stable Rank Normalization for Improved Generalization in Neural Networks and GANscited by 41
  14. 2019Spectral Regularization for Combating Mode Collapse in GANscited by 31
  15. 2020T-Basis: a Compact Representation for Neural Networkscited by 21
  16. 2022Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimescited by 19
  17. 2022TTOpt: A Maximum Volume Quantized Tensor Train-based Optimization and its Application to Reinforcement Learningcited by 15
  18. 2022TT-NF: Tensor Train Neural Fieldscited by 7
  19. 2021Spectral Tensor Train Parameterization of Deep Learning Layerscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 2; 2026: 3.

9 papers.

  1. 2025An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architecturescited by 8
  2. 2024Spectrum Extraction and Clipping for Implicitly Linear Layerscited by 7
  3. 2024Tight and Efficient Upper Bound on Spectral Norm of Convolutional Layerscited by 7
  4. 2024Spectral regularization for adversarially-robust representation learningcited by 2
  5. 2024LOTOS: Layer-wise Orthogonalization for Training Robust Ensemblescited by 1
  6. 2026Can Spectral-Clipping Enable Better Learning While Forgetting Less for Low-Rank Adaptation?cited by 1
  7. 2025Disentangled and Distilled Encoder for Out-of-Distribution Reasoning with Rademacher Guaranteescited by 0
  8. 2026Orthogonium : A Unified, Efficient Library of Orthogonal and 1-Lipschitz Building Blockscited by 0
  9. 2026Detecting Adversarial Data via Provable Adversarial Noise Amplificationcited by 0