# Improved Bilinear Pooling with CNNs

arXiv: https://arxiv.org/abs/1707.06772 · HTML: https://mlsanity.com/paper/1707.06772/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jul 2017.

## Built on by (most cited first, 29 of 29)

- [Fine-Grained Image Analysis with Deep Learning: A Survey](https://mlsanity.com/paper/2111.06119/) (2021)
- [Feature Fusion Vision Transformer for Fine-Grained Visual Categorization](https://mlsanity.com/paper/2107.02341/) (2021)
- [LOT: Layer-wise Orthogonal Training on Improving ℓ2 Certified Robustness](https://mlsanity.com/paper/2210.11620/) (2022)
- [Why Approximate Matrix Square Root Outperforms Accurate SVD in Global Covariance Pooling?](https://mlsanity.com/paper/2105.02498/) (2021)
- [Robust Differentiable SVD](https://mlsanity.com/paper/2104.03821/) (2021)
- [Fast Differentiable Matrix Square Root and Inverse Square Root](https://mlsanity.com/paper/2201.12543/) (2022)
- [High-order Tensor Pooling with Attention for Action Recognition](https://mlsanity.com/paper/2110.05216/) (2021)
- [Fast Differentiable Matrix Square Root](https://mlsanity.com/paper/2201.08663/) (2022)
- [On the Eigenvalues of Global Covariance Pooling for Fine-grained Visual Recognition](https://mlsanity.com/paper/2205.13282/) (2022)
- [Tuning Pre-trained Model via Moment Probing](https://mlsanity.com/paper/2307.11342/) (2023)
- [Understanding Matrix Function Normalizations in Covariance Pooling through the Lens of Riemannian Geometry](https://mlsanity.com/paper/2407.10484/) (2024)
- [Fine-Grained Visual Classification using Self Assessment Classifier](https://mlsanity.com/paper/2205.10529/) (2022)
- [Learning Partial Correlation based Deep Visual Representation for Image Classification](https://mlsanity.com/paper/2304.11597/) (2023)
- [SoT: Delving Deeper into Classification Head for Transformer](https://mlsanity.com/paper/2104.10935/) (2021)
- [Self-Supervised Learning to Guide Scientifically Relevant Categorization of Martian Terrain Images](https://mlsanity.com/paper/2204.09854/) (2022)
- [Fine-Grained Adversarial Semi-supervised Learning](https://mlsanity.com/paper/2110.05848/) (2021)
- [Fine-grained Activities of People Worldwide](https://mlsanity.com/paper/2207.05182/) (2022)
- [Attention to the Burstiness in Visual Prompt Tuning!](https://mlsanity.com/paper/2506.22908/) (2025)
- [A Compositional Feature Embedding and Similarity Metric for Ultra-Fine-Grained Visual Categorization](https://mlsanity.com/paper/2109.12380/) (2021)
- [Batch-efficient EigenDecomposition for Small and Medium Matrices](https://mlsanity.com/paper/2207.04228/) (2022)
- [Kernel function impact on convolutional neural networks](https://mlsanity.com/paper/2302.10266/) (2023)
- [Enhancing Martian Terrain Recognition with Deep Constrained Clustering](https://mlsanity.com/paper/2503.17633/) (2025)
- [Learning a Discriminant Latent Space with Neural Discriminant Analysis](https://mlsanity.com/paper/2107.06209/) (2021)
- [Mask-Guided Feature Extraction and Augmentation for Ultra-Fine-Grained Visual Categorization](https://mlsanity.com/paper/2109.07755/) (2021)
- [Feature Correlation Aggregation: on the Path to Better Graph Neural Networks](https://mlsanity.com/paper/2109.09300/) (2021)
- [A Novel Structured Natural Gradient Descent for Deep Learning](https://mlsanity.com/paper/2109.10100/) (2021)
- [A Deeper Look into Second-Order Feature Aggregation for LiDAR Place Recognition](https://mlsanity.com/paper/2409.15919/) (2024)
- [Conditional Temporal Neural Processes with Covariance Loss](https://mlsanity.com/paper/2504.00794/) (2025)
- [Voronoi-based Second-order Descriptor with Whitened Metric in LiDAR Place Recognition](https://mlsanity.com/paper/2603.14974/) (2026)

## Abstract (arXiv)

Bilinear pooling of Convolutional Neural Network (CNN) features [22, 23], and their compact variants [10], have been shown to be effective at fine-grained recognition, scene categorization, texture recognition, and visual question-answering tasks among others. The resulting representation captures second-order statistics of convolutional features in a translationally invariant manner. In this paper we investigate various ways of normalizing these statistics to improve their representation power. In particular we find that the matrix square-root normalization offers significant improvements and outperforms alternative schemes such as the matrix logarithm normalization when combined with elementwise square-root and l2 normalization. This improves the accuracy by 2-3% on a range of fine-grained recognition datasets leading to a new state of the art. We also investigate how the accuracy of matrix function computations effect network training and evaluation. In particular we compare against a technique for estimating matrix square-root gradients via solving a Lyapunov equation that is more numerically accurate than computing gradients via a Singular Value Decomposition (SVD). We find that while SVD gradients are numerically inaccurate the overall effect on the final accuracy is negligible once boundary cases are handled carefully. We present an alternative scheme for computing gradients that is faster and yet it offers improvements over the baseline model. Finally we show that the matrix square-root computed approximately using a few Newton iterations is just as accurate for the classification task but allows an order-of-magnitude faster GPU implementation compared to SVD decomposition.