mlsanityData to 7 Oct 2026

Existence, Stability and Scalability of Orthogonal Convolutional Neural Networks

arXiv 2108.05623 ↗First posted 12 Aug 2021math.ST cs.AI stat.THcited by 16

Not on the map. Why?

Abstract (arXiv)

Imposing orthogonality on the layers of neural networks is known to facilitate the learning by limiting the exploding/vanishing of the gradient; decorrelate the features; improve the robustness. This paper studies the theoretical properties of orthogonal convolutional layers.We establish necessary and sufficient conditions on the layer architecture guaranteeing the existence of an orthogonal convolutional transform. The conditions prove that orthogonal convolutional transforms exist for almost all architectures used in practice for 'circular' padding.We also exhibit limitations with 'valid' boundary conditions and 'same' boundary conditions with zero-padding.Recently, a regularization term imposing the orthogonality of convolutional layers has been proposed, and impressive empirical results have been obtained in different applications (Wang et al. 2020).The second motivation of the present paper is to specify the theory behind this.We make the link between this regularization term and orthogonality measures. In doing so, we show that this regularization strategy is stable with respect to numerical and optimization errors and that, in the presence of small errors and when the size of the signal/image is large, the convolutional layers remain close to isometric.The theoretical results are confirmed with experiments and the landscape of the regularization term is studied. Experiments on real data sets show that when orthogonality is used to enforce robustness, the parameter multiplying the regularization termcan be used to tune a tradeoff between accuracy and orthogonality, for the benefit of both accuracy and robustness.Altogether, the study guarantees that the regularization proposed in Wang et al. (2020) is an efficient, flexible and stable numerical strategy to learn orthogonal convolutional layers.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 47 of its 63 references that are in the map. First posted on arXiv 12 Aug 2021.

Builds on

47 of 63 references.

References · 47

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  8. 2013Intriguing properties of neural networkscited by 5,114
  9. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  10. 2013Auto-Encoding Variational Bayescited by 3,980
  11. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  12. 2017Improved Training of Wasserstein GANscited by 2,223
35 more
  1. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  2. 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 774
  3. 2017Towards Principled Methods for Training Generative Adversarial Networkscited by 460
  4. 2018Invariant Information Clustering for Unsupervised Image Classification and Segmentationcited by 454
  5. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 413
  6. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 311
  7. 2015Unitary Evolution Recurrent Neural Networkscited by 291
  8. 2017Parseval Networks: Improving Robustness to Adversarial Examplescited by 288
  9. 2018Sorting out Lipschitz function approximationcited by 211
  10. 2018Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networkscited by 193
  11. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  12. 2020The Lipschitz Constant of Self-Attentioncited by 171
  13. 2018Scaling provable adversarial defensescited by 141
  14. 2019Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Groupcited by 133
  15. 2019Orthogonal Convolutional Neural Networkscited by 122
  16. 2016Robust Large Margin Deep Neural Networkscited by 117
  17. 2021Orthogonalizing Convolutional Layers with the Cayley Transformcited by 105
  18. 2018The Singular Values of Convolutional Layerscited by 95
  19. 2017Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networkscited by 92
  20. 2016Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Unitscited by 86
  21. 2019Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networkscited by 86
  22. 2020Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transformcited by 79
  23. 2019Orthogonal Deep Neural Networkscited by 72
  24. 2018Generalizable Adversarial Training via Spectral Normalizationcited by 68
  25. 2021Skew Orthogonal Convolutionscited by 67
  26. 2018Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?cited by 62
  27. 2017All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulationcited by 58
  28. 2020Deep Isometric Learning for Visual Recognitioncited by 38
  29. 2020Controllable Orthogonalization in Training DNNscited by 30
  30. 2021Pay attention to your loss: understanding misconceptions about 1-Lipschitz neural networkscited by 30
  31. 2020Achieving robustness in classification using optimal transport with hinge regularizationcited by 27
  32. 2018L2-Nonexpansive Neural Networkscited by 21
  33. 2021Orthogonal Ensemble Networks for Biomedical Image Segmentationcited by 16
  34. 2020Separating the Effects of Batch Normalization on CNN Training Speed and Stability Using Classical Adaptive Filter Theorycited by 4
  35. 2019Approximated Orthonormal Normalisation in Training Neural Networkscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 4; 2024: 7; 2025: 3; 2026: 1.

16 papers.

Sort:

By area:

  1. 2023DP-SGD Without Clipping: The Lipschitz Neural Network Waycited by 11
  2. 2023Robust One-Class Classification with Signed Distance Function using 1-Lipschitz Neural Networkscited by 10
  3. 2025An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architecturescited by 8
  4. 2024Spectrum Extraction and Clipping for Implicitly Linear Layerscited by 7
  5. 2022On the explainable properties of 1-Lipschitz Neural Networks: An Optimal Transport Perspectivecited by 6
  6. 2024Spectrum-Aware Parameter Efficient Fine-Tuning for Diffusion Modelscited by 5
  7. 2025Enhancing Certified Robustness via Block Reflector Orthogonal Layers and Logit Annealing Losscited by 5
  8. 2024Spectral Norm of Convolutional Layers with Circular and Zero Paddingscited by 4
  9. 2025Deep Polynomial Chaos Expansioncited by 3
  10. 2023Convergence Analysis for Learning Orthonormal Deep Linear Neural Networkscited by 1
  11. 2024Local convergence of simultaneous min-max algorithms to differential equilibrium on Riemannian manifoldcited by 1
  12. 2023Convex optimization over a probability simplexcited by 0
4 more
  1. 2024Towards stable training of parallel continual learningcited by 0
  2. 2024Adversarial Vulnerability as a Consequence of On-Manifold Inseparibilitycited by 0
  3. 2024Feature Map Similarity Reduction in Convolutional Neural Networkscited by 0
  4. 2026Orthogonium : A Unified, Efficient Library of Orthogonal and 1-Lipschitz Building Blockscited by 0