mlsanityData to 7 Oct 2026

Improving Covariance Conditioning of the SVD Meta-layer by Orthogonality

arXiv 2207.02119 ↗First posted 5 Jul 2022cs.CV cs.LG cs.NAcited by 4

Not on the map. Why?

Abstract (arXiv)

Inserting an SVD meta-layer into neural networks is prone to make the covariance ill-conditioned, which could harm the model in the training stability and generalization abilities. In this paper, we systematically study how to improve the covariance conditioning by enforcing orthogonality to the Pre-SVD layer. Existing orthogonal treatments on the weights are first investigated. However, these techniques can improve the conditioning but would hurt the performance. To avoid such a side effect, we propose the Nearest Orthogonal Gradient (NOG) and Optimal Learning Rate (OLR). The effectiveness of our methods is validated in two applications: decorrelated Batch Normalization (BN) and Global Covariance Pooling (GCP). Extensive experiments on visual recognition demonstrate that our methods can simultaneously improve the covariance conditioning and generalization. Moreover, the combinations with orthogonal weight can further boost the performances.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 45 of its 63 references that are in the map. First posted on arXiv 5 Jul 2022.

Builds on

45 of 63 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2022Generative Adversarial Networkscited by 5,603
  3. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  4. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  5. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  6. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  7. 2017Universal Style Transfer via Feature Transformscited by 352
  8. 2016DSAC - Differentiable RANSAC for Camera Localizationcited by 264
  9. 2017Improved Texture Networks: Maximizing Quality and Diversity in Feed-forward Stylization and Texture Synthesiscited by 215
  10. 2021RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whiteningcited by 213
  11. 2018Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networkscited by 193
  12. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
33 more
  1. 2021SOFT: Softmax-free Transformer with Linear Complexitycited by 128
  2. 2015All you need is a good initcited by 124
  3. 2019Orthogonal Convolutional Neural Networkscited by 122
  4. 2021Orthogonalizing Convolutional Layers with the Cayley Transformcited by 105
  5. 2018The Singular Values of Convolutional Layerscited by 95
  6. 2019Iterative Normalization: Beyond Standardization towards Efficient Whiteningcited by 92
  7. 2018Decorrelated Batch Normalizationcited by 83
  8. 2021Is Attention Better Than Matrix Decomposition?cited by 75
  9. 2014PCANet: A Simple Deep Learning Baseline for Image Classification?cited by 70
  10. 2021Skew Orthogonal Convolutionscited by 67
  11. 2017Towards Faster Training of Global Covariance Pooling Networks by Iterative Matrix Square Root Normalizationcited by 65
  12. 2018Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?cited by 62
  13. 2018Image-to-Image Translation via Group-wise Deep Whitening-and-Coloring Transformationcited by 49
  14. 2017Is Second-order Information Helpful for Large-scale Visual Recognition?cited by 44
  15. 2019Deep CNNs Meet Global Covariance Pooling: Better Representation and Generalizationcited by 42
  16. 2019Diversified Arbitrary Style Transfer via Deep Feature Perturbationcited by 42
  17. 2016Regularizing CNNs with Locally Constrained Decorrelationscited by 41
  18. 2020Deep Isometric Learning for Visual Recognitioncited by 38
  19. 2020Solving the Blind Perspective-n-Point Problem End-To-End With Robust Differentiable Geometric Optimizationcited by 38
  20. 2015Training Deep Networks with Structured Layers by Matrix Backpropagationcited by 28
  21. 2017ADMM-Net: A Deep Learning Approach for Compressive Sensing MRIcited by 28
  22. 2018Whitening and Coloring batch transform for GANscited by 27
  23. 2021Why Approximate Matrix Square Root Outperforms Accurate SVD in Global Covariance Pooling?cited by 24
  24. 2021Robust Differentiable SVDcited by 19
  25. 2020Keep it Simple: Image Statistics Matching for Domain Adaptationcited by 17
  26. 2022Fast Differentiable Matrix Square Root and Inverse Square Rootcited by 16
  27. 2020An Investigation into the Stochasticity of Batch Whiteningcited by 14
  28. 2022Fast Differentiable Matrix Square Rootcited by 14
  29. 2020Group Whitening: Balancing Learning Efficiency and Representational Capacitycited by 13
  30. 2018Complex Unitary Recurrent Neural Networks using Scaled Cayley Transformcited by 12
  31. 2021Temporal-attentive Covariance Pooling Networks for Video Recognitioncited by 11
  32. 2022On the Eigenvalues of Global Covariance Pooling for Fine-grained Visual Recognitioncited by 10
  33. 2020Eigendecomposition-Free Training of Deep Networks for Linear Least-Square Problemscited by 7

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 1; 2024: 2.

4 papers.

  1. 2024Understanding Matrix Function Normalizations in Covariance Pooling through the Lens of Riemannian Geometrycited by 9
  2. 2022Orthogonal SVD Covariance Conditioning and Latent Disentanglementcited by 5
  3. 2023Householder Projector for Unsupervised Latent Semantics Discoverycited by 5
  4. 2024OTLRM: Orthogonal Learning-based Low-Rank Metric for Multi-Dimensional Inverse Problemscited by 5