# Why Approximate Matrix Square Root Outperforms Accurate SVD in Global Covariance Pooling?

arXiv: https://arxiv.org/abs/2105.02498 · HTML: https://mlsanity.com/paper/2105.02498/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 31 references that are in the map.
- First posted on arXiv 6 May 2021.

## Builds on (most cited first, 13 of 13)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Bilinear CNNs for Fine-grained Visual Recognition](https://mlsanity.com/paper/1504.07889/) (2015)
- [Decorrelated Batch Normalization](https://mlsanity.com/paper/1804.08450/) (2018)
- [Towards Faster Training of Global Covariance Pooling Networks by Iterative Matrix Square Root Normalization](https://mlsanity.com/paper/1712.01034/) (2017)
- [Image-to-Image Translation via Group-wise Deep Whitening-and-Coloring Transformation](https://mlsanity.com/paper/1812.09912/) (2018)
- [Is Second-order Information Helpful for Large-scale Visual Recognition?](https://mlsanity.com/paper/1703.08050/) (2017)
- [Deep CNNs Meet Global Covariance Pooling: Better Representation and Generalization](https://mlsanity.com/paper/1904.06836/) (2019)
- [Training Deep Networks with Structured Layers by Matrix Backpropagation](https://mlsanity.com/paper/1509.07838/) (2015)
- [Improved Bilinear Pooling with CNNs](https://mlsanity.com/paper/1707.06772/) (2017)
- [Backpropagation-Friendly Eigendecomposition](https://mlsanity.com/paper/1906.09023/) (2019)
- [Dynamic Label Graph Matching for Unsupervised Video Re-Identification](https://mlsanity.com/paper/1709.09297/) (2017)
- [An Investigation into the Stochasticity of Batch Whitening](https://mlsanity.com/paper/2003.12327/) (2020)
- [What Deep CNNs Benefit from Global Covariance Pooling: An Optimization Perspective](https://mlsanity.com/paper/2003.11241/) (2020)

## Built on by (most cited first, 24 of 24)

- [Fast Differentiable Matrix Square Root and Inverse Square Root](https://mlsanity.com/paper/2201.12543/) (2022)
- [Fast Differentiable Matrix Square Root](https://mlsanity.com/paper/2201.08663/) (2022)
- [Adaptive Log-Euclidean Metrics for SPD Matrix Learning](https://mlsanity.com/paper/2303.15477/) (2023)
- [Self-Consistency Training for Density-Functional-Theory Hamiltonian Prediction](https://mlsanity.com/paper/2403.09560/) (2024)
- [Riemannian Multinomial Logistics Regression for SPD Neural Networks](https://mlsanity.com/paper/2305.11288/) (2023)
- [On the Eigenvalues of Global Covariance Pooling for Fine-grained Visual Recognition](https://mlsanity.com/paper/2205.13282/) (2022)
- [Understanding Matrix Function Normalizations in Covariance Pooling through the Lens of Riemannian Geometry](https://mlsanity.com/paper/2407.10484/) (2024)
- [RMLR: Extending Multinomial Logistic Regression into General Geometries](https://mlsanity.com/paper/2409.19433/) (2024)
- [Learning Partial Correlation based Deep Visual Representation for Image Classification](https://mlsanity.com/paper/2304.11597/) (2023)
- [Convolutional Fine-Grained Classification with Self-Supervised Target Relation Regularization](https://mlsanity.com/paper/2208.01997/) (2022)
- [Grouping-matrix based Graph Pooling with Adaptive Number of Clusters](https://mlsanity.com/paper/2209.02939/) (2022)
- [Orthogonal SVD Covariance Conditioning and Latent Disentanglement](https://mlsanity.com/paper/2212.05599/) (2022)
- [Improving Covariance Conditioning of the SVD Meta-layer by Orthogonality](https://mlsanity.com/paper/2207.02119/) (2022)
- [Spectral Norm of Convolutional Layers with Circular and Zero Paddings](https://mlsanity.com/paper/2402.00240/) (2024)
- [PRISM: Distribution-free Adaptive Computation of Matrix Functions for Accelerating Neural Network Training](https://mlsanity.com/paper/2601.22137/) (2026)
- [KrADagrad: Kronecker Approximation-Domination Gradient Preconditioned Stochastic Optimization](https://mlsanity.com/paper/2305.19416/) (2023)
- [Batch-efficient EigenDecomposition for Small and Medium Matrices](https://mlsanity.com/paper/2207.04228/) (2022)
- [Adversarial Defence without Adversarial Defence: Enhancing Language Model Robustness via Instance-level Principal Component Removal](https://mlsanity.com/paper/2507.21750/) (2025)
- [MatRL: Provably Generalizable Iterative Algorithm Discovery via Monte-Carlo Tree Search](https://mlsanity.com/paper/2507.03833/) (2025)
- [Automatic Classification of Sleep Stages from EEG Signals Using Riemannian Metrics and Transformer Networks](https://mlsanity.com/paper/2410.19819/) (2024)
- [Open Vocabulary Compositional Explanations for Neuron Alignment](https://mlsanity.com/paper/2511.20931/) (2025)
- [Shallow Neural Networks Learn Low-Degree Spherical Polynomials with Feature Learning by Learnable Channel Attention](https://mlsanity.com/paper/2512.20562/) (2025)
- [Learning to Discover Iterative Spectral Algorithms](https://mlsanity.com/paper/2602.09530/) (2026)
- [Orthogonal Polynomial Approximation for Matrix Log Normalization in Global Covariance Pooling](https://mlsanity.com/paper/2608.19021/) (2026)

## Abstract (arXiv)

Global covariance pooling (GCP) aims at exploiting the second-order statistics of the convolutional feature. Its effectiveness has been demonstrated in boosting the classification performance of Convolutional Neural Networks (CNNs). Singular Value Decomposition (SVD) is used in GCP to compute the matrix square root. However, the approximate matrix square root calculated using Newton-Schulz iteration citeli2018towards outperforms the accurate one computed via SVD citeli2017second. We empirically analyze the reason behind the performance gap from the perspectives of data precision and gradient smoothness. Various remedies for computing smooth SVD gradients are investigated. Based on our observation and analyses, a hybrid training protocol is proposed for SVD-based GCP meta-layers such that competitive performances can be achieved against Newton-Schulz iteration. Moreover, we propose a new GCP meta-layer that uses SVD in the forward pass, and Pad'e Approximants in the backward propagation to compute the gradients. The proposed meta-layer has been integrated into different CNN models and achieves state-of-the-art performances on both large-scale and fine-grained datasets.