# Deep CNNs Meet Global Covariance Pooling: Better Representation and Generalization

arXiv: https://arxiv.org/abs/1904.06836 · HTML: https://mlsanity.com/paper/1904.06836/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Apr 2019.

## Built on by (most cited first, 30 of 42)

- [Fine-Grained Image Analysis with Deep Learning: A Survey](https://mlsanity.com/paper/2111.06119/) (2021)
- [Joint Distribution Matters: Deep Brownian Distance Covariance for Few-Shot Classification](https://mlsanity.com/paper/2204.04567/) (2022)
- [Why Approximate Matrix Square Root Outperforms Accurate SVD in Global Covariance Pooling?](https://mlsanity.com/paper/2105.02498/) (2021)
- [Distribution Knowledge Embedding for Graph Pooling](https://mlsanity.com/paper/2109.14333/) (2021)
- [WaveMix: A Resource-efficient Neural Network for Image Analysis](https://mlsanity.com/paper/2205.14375/) (2022)
- [Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation](https://mlsanity.com/paper/2412.08139/) (2024)
- [Fast Differentiable Matrix Square Root and Inverse Square Root](https://mlsanity.com/paper/2201.12543/) (2022)
- [Fast Differentiable Matrix Square Root](https://mlsanity.com/paper/2201.08663/) (2022)
- [Temporal-attentive Covariance Pooling Networks for Video Recognition](https://mlsanity.com/paper/2110.14381/) (2021)
- [Riemannian Multinomial Logistics Regression for SPD Neural Networks](https://mlsanity.com/paper/2305.11288/) (2023)
- [On the Eigenvalues of Global Covariance Pooling for Fine-grained Visual Recognition](https://mlsanity.com/paper/2205.13282/) (2022)
- [SLOSH: Set LOcality Sensitive Hashing via Sliced-Wasserstein Embeddings](https://mlsanity.com/paper/2112.05872/) (2021)
- [Understanding Matrix Function Normalizations in Covariance Pooling through the Lens of Riemannian Geometry](https://mlsanity.com/paper/2407.10484/) (2024)
- [RMLR: Extending Multinomial Logistic Regression into General Geometries](https://mlsanity.com/paper/2409.19433/) (2024)
- [Bi-directional Object-context Prioritization Learning for Saliency Ranking](https://mlsanity.com/paper/2203.09416/) (2022)
- [Learning Partial Correlation based Deep Visual Representation for Image Classification](https://mlsanity.com/paper/2304.11597/) (2023)
- [SoT: Delving Deeper into Classification Head for Transformer](https://mlsanity.com/paper/2104.10935/) (2021)
- [Convolutional Fine-Grained Classification with Self-Supervised Target Relation Regularization](https://mlsanity.com/paper/2208.01997/) (2022)
- [Orthogonal SVD Covariance Conditioning and Latent Disentanglement](https://mlsanity.com/paper/2212.05599/) (2022)
- [Fine-Grained Adversarial Semi-supervised Learning](https://mlsanity.com/paper/2110.05848/) (2021)
- [Improving Covariance Conditioning of the SVD Meta-layer by Orthogonality](https://mlsanity.com/paper/2207.02119/) (2022)
- [Fast and Stable Riemannian Metrics on SPD Manifolds via Cholesky Product Geometry](https://mlsanity.com/paper/2407.02607/) (2024)
- [PRISM: Distribution-free Adaptive Computation of Matrix Functions for Accelerating Neural Network Training](https://mlsanity.com/paper/2601.22137/) (2026)
- [A Novel Image Classification Framework Based on Variational Quantum Algorithms](https://mlsanity.com/paper/2312.07932/) (2023)
- [FineFACE: Fair Facial Attribute Classification Leveraging Fine-grained Features](https://mlsanity.com/paper/2408.16881/) (2024)
- [Sheaf Neural Networks on SPD Manifolds: Second-Order Geometric Representation Learning](https://mlsanity.com/paper/2604.20308/) (2026)
- [GAN for Vision, KG for Relation: a Two-stage Deep Network for Zero-shot Action Recognition](https://mlsanity.com/paper/2105.11789/) (2021)
- [Batch-efficient EigenDecomposition for Small and Medium Matrices](https://mlsanity.com/paper/2207.04228/) (2022)
- [Equivariant vs. Invariant Layers: A Comparison of Backbone and Pooling for Point Cloud Classification](https://mlsanity.com/paper/2306.05553/) (2023)
- [NOAH: Learning Pairwise Object Category Attentions for Image Classification](https://mlsanity.com/paper/2402.02377/) (2024)

## Abstract (arXiv)

Compared with global average pooling in existing deep convolutional neural networks (CNNs), global covariance pooling can capture richer statistics of deep features, having potential for improving representation and generalization abilities of deep CNNs. However, integration of global covariance pooling into deep CNNs brings two challenges: (1) robust covariance estimation given deep features of high dimension and small sample size; (2) appropriate usage of geometry of covariances. To address these challenges, we propose a global Matrix Power Normalized COVariance (MPN-COV) Pooling. Our MPN-COV conforms to a robust covariance estimator, very suitable for scenario of high dimension and small sample size. It can also be regarded as Power-Euclidean metric between covariances, effectively exploiting their geometry. Furthermore, a global Gaussian embedding network is proposed to incorporate first-order statistics into MPN-COV. For fast training of MPN-COV networks, we implement an iterative matrix square root normalization, avoiding GPU unfriendly eigen-decomposition inherent in MPN-COV. Additionally, progressive 1x1 convolutions and group convolution are introduced to compress covariance representations. The proposed methods are highly modular, readily plugged into existing deep CNNs. Extensive experiments are conducted on large-scale object classification, scene categorization, fine-grained visual recognition and texture classification, showing our methods outperform the counterparts and obtain state-of-the-art performance.