# Towards Faster Training of Global Covariance Pooling Networks by Iterative Matrix Square Root Normalization

arXiv: https://arxiv.org/abs/1712.01034 · HTML: https://mlsanity.com/paper/1712.01034/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Dec 2017.

## Built on by (most cited first, 30 of 65)

- [Fine-Grained Image Analysis with Deep Learning: A Survey](https://mlsanity.com/paper/2111.06119/) (2021)
- [Feature Fusion Vision Transformer for Fine-Grained Visual Categorization](https://mlsanity.com/paper/2107.02341/) (2021)
- [Webly Supervised Fine-Grained Recognition: Benchmark Datasets and An Approach](https://mlsanity.com/paper/2108.02399/) (2021)
- [Context-aware Attentional Pooling (CAP) for Fine-grained Visual Classification](https://mlsanity.com/paper/2101.06635/) (2021)
- [Why Approximate Matrix Square Root Outperforms Accurate SVD in Global Covariance Pooling?](https://mlsanity.com/paper/2105.02498/) (2021)
- [Vector-valued Distance and Gyrocalculus on the Space of Symmetric Positive Definite Matrices](https://mlsanity.com/paper/2110.13475/) (2021)
- [SWAN: SGD with Normalization and Whitening Enables Stateless LLM Training](https://mlsanity.com/paper/2412.13148/) (2024)
- [Robust Differentiable SVD](https://mlsanity.com/paper/2104.03821/) (2021)
- [Distribution Knowledge Embedding for Graph Pooling](https://mlsanity.com/paper/2109.14333/) (2021)
- [Fast Differentiable Matrix Square Root and Inverse Square Root](https://mlsanity.com/paper/2201.12543/) (2022)
- [Learning Gabor Texture Features for Fine-Grained Recognition](https://mlsanity.com/paper/2308.05396/) (2023)
- [Fast Differentiable Matrix Square Root](https://mlsanity.com/paper/2201.08663/) (2022)
- [Temporal-attentive Covariance Pooling Networks for Video Recognition](https://mlsanity.com/paper/2110.14381/) (2021)
- [OH-Former: Omni-Relational High-Order Transformer for Person Re-Identification](https://mlsanity.com/paper/2109.11159/) (2021)
- [On the Eigenvalues of Global Covariance Pooling for Fine-grained Visual Recognition](https://mlsanity.com/paper/2205.13282/) (2022)
- [Online Continual Learning for Robust Indoor Object Recognition](https://mlsanity.com/paper/2307.09827/) (2023)
- [Tuning Pre-trained Model via Moment Probing](https://mlsanity.com/paper/2307.11342/) (2023)
- [Online Continual Learning in Keyword Spotting for Low-Resource Devices via Pooling High-Order Temporal Statistics](https://mlsanity.com/paper/2307.12660/) (2023)
- [Understanding Matrix Function Normalizations in Covariance Pooling through the Lens of Riemannian Geometry](https://mlsanity.com/paper/2407.10484/) (2024)
- [Gradient Multi-Normalization for Stateless and Scalable LLM Training](https://mlsanity.com/paper/2502.06742/) (2025)
- [Onfocus Detection: Identifying Individual-Camera Eye Contact from Unconstrained Images](https://mlsanity.com/paper/2103.15307/) (2021)
- [Learning Partial Correlation based Deep Visual Representation for Image Classification](https://mlsanity.com/paper/2304.11597/) (2023)
- [SPD Matrix Learning for Neuroimaging Analysis: Perspectives, Methods, and Challenges](https://mlsanity.com/paper/2504.18882/) (2025)
- [Self-Supervised AI-Generated Image Detection: A Camera Metadata Perspective](https://mlsanity.com/paper/2512.05651/) (2025)
- [SoT: Delving Deeper into Classification Head for Transformer](https://mlsanity.com/paper/2104.10935/) (2021)
- [Convolutional Fine-Grained Classification with Self-Supervised Target Relation Regularization](https://mlsanity.com/paper/2208.01997/) (2022)
- [Orthogonal SVD Covariance Conditioning and Latent Disentanglement](https://mlsanity.com/paper/2212.05599/) (2022)
- [Fine-Grained Adversarial Semi-supervised Learning](https://mlsanity.com/paper/2110.05848/) (2021)
- [Improving Covariance Conditioning of the SVD Meta-layer by Orthogonality](https://mlsanity.com/paper/2207.02119/) (2022)
- [Fast and Stable Riemannian Metrics on SPD Manifolds via Cholesky Product Geometry](https://mlsanity.com/paper/2407.02607/) (2024)

## Abstract (arXiv)

Global covariance pooling in convolutional neural networks has achieved impressive improvement over the classical first-order pooling. Recent works have shown matrix square root normalization plays a central role in achieving state-of-the-art performance. However, existing methods depend heavily on eigendecomposition (EIG) or singular value decomposition (SVD), suffering from inefficient training due to limited support of EIG and SVD on GPU. Towards addressing this problem, we propose an iterative matrix square root normalization method for fast end-to-end training of global covariance pooling networks. At the core of our method is a meta-layer designed with loop-embedded directed graph structure. The meta-layer consists of three consecutive nonlinear structured layers, which perform pre-normalization, coupled matrix iteration and post-compensation, respectively. Our method is much faster than EIG or SVD based ones, since it involves only matrix multiplications, suitable for parallel implementation on GPU. Moreover, the proposed network with ResNet architecture can converge in much less epochs, further accelerating network training. On large-scale ImageNet, we achieve competitive performance superior to existing counterparts. By finetuning our models pre-trained on ImageNet, we establish state-of-the-art results on three challenging fine-grained benchmarks. The source code and network models will be available at http://www.peihuali.org/iSQRT-COV