# Higher-order Pooling of CNN Features via Kernel Linearization for Action Recognition

arXiv: https://arxiv.org/abs/1701.05432 · HTML: https://mlsanity.com/paper/1701.05432/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jan 2017.

## Built on by (most cited first, 4 of 4)

- [3Mformer: Multi-order Multi-mode Transformer for Skeletal Action Recognition](https://mlsanity.com/paper/2303.14474/) (2023)
- [High-order Tensor Pooling with Attention for Action Recognition](https://mlsanity.com/paper/2110.05216/) (2021)
- [Feature Hallucination for Self-supervised Action Recognition](https://mlsanity.com/paper/2506.20342/) (2025)
- [Learning Log-Determinant Divergences for Positive Definite Matrices](https://mlsanity.com/paper/2104.06461/) (2021)

## Abstract (arXiv)

Most successful deep learning algorithms for action recognition extend models designed for image-based tasks such as object recognition to video. Such extensions are typically trained for actions on single video frames or very short clips, and then their predictions from sliding-windows over the video sequence are pooled for recognizing the action at the sequence level. Usually this pooling step uses the first-order statistics of frame-level action predictions. In this paper, we explore the advantages of using higher-order correlations; specifically, we introduce Higher-order Kernel (HOK) descriptors generated from the late fusion of CNN classifier scores from all the frames in a sequence. To generate these descriptors, we use the idea of kernel linearization. Specifically, a similarity kernel matrix, which captures the temporal evolution of deep classifier scores, is first linearized into kernel feature maps. The HOK descriptors are then generated from the higher-order co-occurrences of these feature maps, and are then used as input to a video-level classifier. We provide experiments on two fine-grained action recognition datasets and show that our scheme leads to state-of-the-art results.