# High-order Tensor Pooling with Attention for Action Recognition

arXiv: https://arxiv.org/abs/2110.05216 · HTML: https://mlsanity.com/paper/2110.05216/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 66 references that are in the map.
- First posted on arXiv 11 Oct 2021.

## Builds on (most cited first, 19 of 19)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Spatiotemporal Residual Networks for Video Action Recognition](https://mlsanity.com/paper/1611.02155/) (2016)
- [Attentional Pooling for Action Recognition](https://mlsanity.com/paper/1711.01467/) (2017)
- [Improved Bilinear Pooling with CNNs](https://mlsanity.com/paper/1707.06772/) (2017)
- [A Comparative Review of Recent Kinect-based Action Recognition Algorithms](https://mlsanity.com/paper/1906.09955/) (2019)
- [Tensor Representations for Action Recognition](https://mlsanity.com/paper/2012.14371/) (2020)
- [Power Normalizations in Fine-grained Image, Few-shot Image and Graph Classification](https://mlsanity.com/paper/2012.13975/) (2020)
- [Self-supervising Action Recognition by Statistical Moment and Subspace Descriptors](https://mlsanity.com/paper/2001.04627/) (2020)
- [Fusing Higher-order Features in Graph Neural Networks for Skeleton-based Action Recognition](https://mlsanity.com/paper/2105.01563/) (2021)
- [Domain Adaptation by Mixture of Alignments of Second- or Higher-Order Scatter Tensors](https://mlsanity.com/paper/1611.08195/) (2016)
- [Loss Switching Fusion with Similarity Search for Video Classification](https://mlsanity.com/paper/1906.11465/) (2019)
- [A Deeper Look at Power Normalizations](https://mlsanity.com/paper/1806.09183/) (2018)
- [Non-Linear Temporal Subspace Representations for Activity Recognition](https://mlsanity.com/paper/1803.11064/) (2018)
- [Tensor Representations via Kernel Linearization for Action Recognition from 3D Skeletons (Extended Version)](https://mlsanity.com/paper/1604.00239/) (2016)
- [Extrinsic Methods for Coding and Dictionary Learning on Grassmann Manifolds](https://mlsanity.com/paper/1401.8126/) (2014)
- [Higher-order Pooling of CNN Features via Kernel Linearization for Action Recognition](https://mlsanity.com/paper/1701.05432/) (2017)
- [Local Temporal Bilinear Pooling for Fine-grained Action Parsing](https://mlsanity.com/paper/1812.01922/) (2018)

## Built on by (most cited first, 14 of 14)

- [Advancing Video Anomaly Detection: A Concise Review and a New Dataset](https://mlsanity.com/paper/2402.04857/) (2024)
- [3Mformer: Multi-order Multi-mode Transformer for Skeletal Action Recognition](https://mlsanity.com/paper/2303.14474/) (2023)
- [Temporal-Viewpoint Transportation Plan for Skeletal Few-shot Action Recognition](https://mlsanity.com/paper/2210.16820/) (2022)
- [Do Language Models Understand Time?](https://mlsanity.com/paper/2412.13845/) (2024)
- [Flow Dynamics Correction for Action Recognition](https://mlsanity.com/paper/2310.10059/) (2023)
- [Taylor Videos for Action Recognition](https://mlsanity.com/paper/2402.03019/) (2024)
- [TrackNetV4: Enhancing Fast Sports Object Tracking with Motion Attention Maps](https://mlsanity.com/paper/2409.14543/) (2024)
- [Meet JEANIE: a Similarity Measure for 3D Skeleton Sequences via Temporal-Viewpoint Alignment](https://mlsanity.com/paper/2402.04599/) (2024)
- [When Spatial meets Temporal in Action Recognition](https://mlsanity.com/paper/2411.15284/) (2024)
- [Feature Hallucination for Self-supervised Action Recognition](https://mlsanity.com/paper/2506.20342/) (2025)
- [ConViViT -- A Deep Neural Network Combining Convolutions and Factorized Self-Attention for Human Activity Recognition](https://mlsanity.com/paper/2310.14416/) (2023)
- [Evolving Skeletons: Motion Dynamics in Action Recognition](https://mlsanity.com/paper/2501.02593/) (2025)
- [Subspace Kernel Learning on Tensor Sequences](https://mlsanity.com/paper/2603.19546/) (2026)
- [Trust-Aware Joint Feature-Prediction Discrepancy for Robust Domain Adaptation](https://mlsanity.com/paper/2605.25119/) (2026)

## Abstract (arXiv)

We aim at capturing high-order statistics of feature vectors formed by a neural network, and propose end-to-end second- and higher-order pooling to form a tensor descriptor. Tensor descriptors require a robust similarity measure due to low numbers of aggregated vectors and the burstiness phenomenon, when a given feature appears more/less frequently than statistically expected. The Heat Diffusion Process (HDP) on a graph Laplacian is closely related to the Eigenvalue Power Normalization (EPN) of the covariance/autocorrelation matrix, whose inverse forms a loopy graph Laplacian. We show that the HDP and the EPN play the same role, i.e., to boost or dampen the magnitude of the eigenspectrum thus preventing the burstiness. We equip higher-order tensors with EPN which acts as a spectral detector of higher-order occurrences to prevent burstiness. We also prove that for a tensor of order r built from d dimensional feature descriptors, such a detector gives the likelihood if at least one higher-order occurrence is 'projected' into one of binom(d,r) subspaces represented by the tensor; thus forming a tensor power normalization metric endowed with binom(d,r) such 'detectors'. For experimental contributions, we apply several second- and higher-order pooling variants to action recognition, provide previously not presented comparisons of such pooling variants, and show state-of-the-art results on HMDB-51, YUP++ and MPII Cooking Activities.