# Towards Practical Control of Singular Values of Convolutional Layers

arXiv: https://arxiv.org/abs/2211.13771 · HTML: https://mlsanity.com/paper/2211.13771/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 43 references that are in the map.
- First posted on arXiv 24 Nov 2022.

## Builds on (most cited first, 30 of 31)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Spectrally-normalized margin bounds for neural networks](https://mlsanity.com/paper/1706.08498/) (2017)
- [Regularisation of Neural Networks by Enforcing Lipschitz Continuity](https://mlsanity.com/paper/1804.04368/) (2018)
- [Parseval Networks: Improving Robustness to Adversarial Examples](https://mlsanity.com/paper/1704.08847/) (2017)
- [Tensorizing Neural Networks](https://mlsanity.com/paper/1509.06569/) (2015)
- [Spectral Norm Regularization for Improving the Generalizability of Deep Learning](https://mlsanity.com/paper/1705.10941/) (2017)
- [Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications](https://mlsanity.com/paper/1511.06530/) (2015)
- [Structured Pruning of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1512.08571/) (2015)
- [The Singular Values of Convolutional Layers](https://mlsanity.com/paper/1805.10408/) (2018)
- [Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networks](https://mlsanity.com/paper/1911.00937/) (2019)
- [Skew Orthogonal Convolutions](https://mlsanity.com/paper/2105.11417/) (2021)
- [On the Practicality of Deterministic Epistemic Uncertainty](https://mlsanity.com/paper/2107.00649/) (2021)
- [Ultimate tensorization: compressing convolutional and FC layers alike](https://mlsanity.com/paper/1611.03214/) (2016)
- [Wide Compression: Tensor Ring Nets](https://mlsanity.com/paper/1802.09052/) (2018)
- [Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference](https://mlsanity.com/paper/2007.12540/) (2020)
- [Stable Rank Normalization for Improved Generalization in Neural Networks and GANs](https://mlsanity.com/paper/1906.04659/) (2019)
- [Spectral Regularization for Combating Mode Collapse in GANs](https://mlsanity.com/paper/1908.10999/) (2019)
- [T-Basis: a Compact Representation for Neural Networks](https://mlsanity.com/paper/2007.06631/) (2020)
- [Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimes](https://mlsanity.com/paper/2209.03695/) (2022)
- [TTOpt: A Maximum Volume Quantized Tensor Train-based Optimization and its Application to Reinforcement Learning](https://mlsanity.com/paper/2205.00293/) (2022)
- [TT-NF: Tensor Train Neural Fields](https://mlsanity.com/paper/2209.15529/) (2022)

## Built on by (most cited first, 9 of 9)

- [An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architectures](https://mlsanity.com/paper/2501.07930/) (2025)
- [Spectrum Extraction and Clipping for Implicitly Linear Layers](https://mlsanity.com/paper/2402.16017/) (2024)
- [Tight and Efficient Upper Bound on Spectral Norm of Convolutional Layers](https://mlsanity.com/paper/2409.11859/) (2024)
- [Spectral regularization for adversarially-robust representation learning](https://mlsanity.com/paper/2405.17181/) (2024)
- [LOTOS: Layer-wise Orthogonalization for Training Robust Ensembles](https://mlsanity.com/paper/2410.05136/) (2024)
- [Can Spectral-Clipping Enable Better Learning While Forgetting Less for Low-Rank Adaptation?](https://mlsanity.com/paper/2608.12332/) (2026)
- [Disentangled and Distilled Encoder for Out-of-Distribution Reasoning with Rademacher Guarantees](https://mlsanity.com/paper/2512.10522/) (2025)
- [Orthogonium : A Unified, Efficient Library of Orthogonal and 1-Lipschitz Building Blocks](https://mlsanity.com/paper/2601.13776/) (2026)
- [Detecting Adversarial Data via Provable Adversarial Noise Amplification](https://mlsanity.com/paper/2605.02109/) (2026)

## Abstract (arXiv)

In general, convolutional neural networks (CNNs) are easy to train, but their essential properties, such as generalization error and adversarial robustness, are hard to control. Recent research demonstrated that singular values of convolutional layers significantly affect such elusive properties and offered several methods for controlling them. Nevertheless, these methods present an intractable computational challenge or resort to coarse approximations. In this paper, we offer a principled approach to alleviating constraints of the prior art at the expense of an insignificant reduction in layer expressivity. Our method is based on the tensor-train decomposition; it retains control over the actual singular values of convolutional mappings while providing structurally sparse and hardware-friendly representation. We demonstrate the improved properties of modern CNNs with our method and analyze its impact on the model performance, calibration, and adversarial robustness. The source code is available at: https://github.com/WhiteTeaDragon/practical_svd_conv