# Is Second-order Information Helpful for Large-scale Visual Recognition?

arXiv: https://arxiv.org/abs/1703.08050 · HTML: https://mlsanity.com/paper/1703.08050/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Mar 2017.

## Built on by (most cited first, 30 of 44)

- [Fine-Grained Image Analysis with Deep Learning: A Survey](https://mlsanity.com/paper/2111.06119/) (2021)
- [LoGG3D-Net: Locally Guided Global Descriptor Learning for 3D Place Recognition](https://mlsanity.com/paper/2109.08336/) (2021)
- [Visual Attention Methods in Deep Learning: An In-Depth Survey](https://mlsanity.com/paper/2204.07756/) (2022)
- [GeomNet: A Neural Network Based on Riemannian Geometries of SPD Matrix Space and Cholesky Space for 3D Skeleton-Based Interaction Recognition](https://mlsanity.com/paper/2111.13089/) (2021)
- [Why Approximate Matrix Square Root Outperforms Accurate SVD in Global Covariance Pooling?](https://mlsanity.com/paper/2105.02498/) (2021)
- [A Two-Stage Attentive Network for Single Image Super-Resolution](https://mlsanity.com/paper/2104.10488/) (2021)
- [Robust Differentiable SVD](https://mlsanity.com/paper/2104.03821/) (2021)
- [Distribution Knowledge Embedding for Graph Pooling](https://mlsanity.com/paper/2109.14333/) (2021)
- [Fast Differentiable Matrix Square Root and Inverse Square Root](https://mlsanity.com/paper/2201.12543/) (2022)
- [Surpassing the Human Accuracy: Detecting Gallbladder Cancer from USG Images with Curriculum Learning](https://mlsanity.com/paper/2204.11433/) (2022)
- [Fast Differentiable Matrix Square Root](https://mlsanity.com/paper/2201.08663/) (2022)
- [Temporal-attentive Covariance Pooling Networks for Video Recognition](https://mlsanity.com/paper/2110.14381/) (2021)
- [On the Eigenvalues of Global Covariance Pooling for Fine-grained Visual Recognition](https://mlsanity.com/paper/2205.13282/) (2022)
- [Tuning Pre-trained Model via Moment Probing](https://mlsanity.com/paper/2307.11342/) (2023)
- [Understanding Matrix Function Normalizations in Covariance Pooling through the Lens of Riemannian Geometry](https://mlsanity.com/paper/2407.10484/) (2024)
- [ELA: Efficient Local Attention for Deep Convolutional Neural Networks](https://mlsanity.com/paper/2403.01123/) (2024)
- [Bi-directional Object-context Prioritization Learning for Saliency Ranking](https://mlsanity.com/paper/2203.09416/) (2022)
- [Learning Partial Correlation based Deep Visual Representation for Image Classification](https://mlsanity.com/paper/2304.11597/) (2023)
- [Survey of Automatic Plankton Image Recognition: Challenges, Existing Solutions and Future Perspectives](https://mlsanity.com/paper/2305.11739/) (2023)
- [Adversarial Examples Detection beyond Image Space](https://mlsanity.com/paper/2102.11586/) (2021)
- [SoT: Delving Deeper into Classification Head for Transformer](https://mlsanity.com/paper/2104.10935/) (2021)
- [Convolutional Fine-Grained Classification with Self-Supervised Target Relation Regularization](https://mlsanity.com/paper/2208.01997/) (2022)
- [Orthogonal SVD Covariance Conditioning and Latent Disentanglement](https://mlsanity.com/paper/2212.05599/) (2022)
- [Fair Comparison: Quantifying Variance in Resultsfor Fine-grained Visual Categorization](https://mlsanity.com/paper/2109.03156/) (2021)
- [Improving Covariance Conditioning of the SVD Meta-layer by Orthogonality](https://mlsanity.com/paper/2207.02119/) (2022)
- [PM2: A New Prompting Multi-modal Model Paradigm for Few-shot Medical Image Classification](https://mlsanity.com/paper/2404.08915/) (2024)
- [Classification of Buried Objects from Ground Penetrating Radar Images by using Second Order Deep Learning Models](https://mlsanity.com/paper/2410.07117/) (2024)
- [Sheaf Neural Networks on SPD Manifolds: Second-Order Geometric Representation Learning](https://mlsanity.com/paper/2604.20308/) (2026)
- [Batch-efficient EigenDecomposition for Small and Medium Matrices](https://mlsanity.com/paper/2207.04228/) (2022)
- [FedSPDnet: Geometry-Aware Federated Deep Learning with SPDnet](https://mlsanity.com/paper/2604.22494/) (2026)

## Abstract (arXiv)

By stacking layers of convolution and nonlinearity, convolutional networks (ConvNets) effectively learn from low-level to high-level features and discriminative representations. Since the end goal of large-scale recognition is to delineate complex boundaries of thousands of classes, adequate exploration of feature distributions is important for realizing full potentials of ConvNets. However, state-of-the-art works concentrate only on deeper or wider architecture design, while rarely exploring feature statistics higher than first-order. We take a step towards addressing this problem. Our method consists in covariance pooling, instead of the most commonly used first-order pooling, of high-level convolutional features. The main challenges involved are robust covariance estimation given a small sample of large-dimensional features and usage of the manifold structure of covariance matrices. To address these challenges, we present a Matrix Power Normalized Covariance (MPN-COV) method. We develop forward and backward propagation formulas regarding the nonlinear matrix functions such that MPN-COV can be trained end-to-end. In addition, we analyze both qualitatively and quantitatively its advantage over the well-known Log-Euclidean metric. On the ImageNet 2012 validation set, by combining MPN-COV we achieve over 4%, 3% and 2.5% gains for AlexNet, VGG-M and VGG-16, respectively; integration of MPN-COV into 50-layer ResNet outperforms ResNet-101 and is comparable to ResNet-152. The source code will be available on the project page: http://www.peihuali.org/MPN-COV