# Generalized Rank Pooling for Activity Recognition

arXiv: https://arxiv.org/abs/1704.02112 · HTML: https://mlsanity.com/paper/1704.02112/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Apr 2017.

## Built on by (most cited first, 10 of 10)

- [Unsupervised Discriminative Embedding for Sub-Action Learning in Complex Activities](https://mlsanity.com/paper/2105.00067/) (2021)
- [Revisiting Implicit Differentiation for Learning Problems in Optimal Control](https://mlsanity.com/paper/2310.14468/) (2023)
- [Deep Declarative Dynamic Time Warping for End-to-End Learning of Alignment Paths](https://mlsanity.com/paper/2303.10778/) (2023)
- [Strengthening Skeletal Action Recognizers via Leveraging Temporal Patterns](https://mlsanity.com/paper/2205.14405/) (2022)
- [FT-HID: A Large Scale RGB-D Dataset for First and Third Person Human Interaction Analysis](https://mlsanity.com/paper/2209.10155/) (2022)
- [TAEC: Unsupervised Action Segmentation with Temporal-Aware Embedding and Clustering](https://mlsanity.com/paper/2303.05166/) (2023)
- [Micro-Expression Recognition via Fine-Grained Dynamic Perception](https://mlsanity.com/paper/2509.06015/) (2025)
- [Multi-Step Short-Term Wind Speed Prediction with Rank Pooling and Fast Fourier Transformation](https://mlsanity.com/paper/2211.14434/) (2022)
- [Proving the Potential of Skeleton Based Action Recognition to Automate the Analysis of Manual Processes](https://mlsanity.com/paper/2310.08451/) (2023)
- [Learning Topology-Driven Multi-Subspace Fusion for Grassmannian Deep Network](https://mlsanity.com/paper/2511.08628/) (2025)

## Abstract (arXiv)

Most popular deep models for action recognition split video sequences into short sub-sequences consisting of a few frames; frame-based features are then pooled for recognizing the activity. Usually, this pooling step discards the temporal order of the frames, which could otherwise be used for better recognition. Towards this end, we propose a novel pooling method, generalized rank pooling (GRP), that takes as input, features from the intermediate layers of a CNN that is trained on tiny sub-sequences, and produces as output the parameters of a subspace which (i) provides a low-rank approximation to the features and (ii) preserves their temporal order. We propose to use these parameters as a compact representation for the video sequence, which is then used in a classification setup. We formulate an objective for computing this subspace as a Riemannian optimization problem on the Grassmann manifold, and propose an efficient conjugate gradient scheme for solving it. Experiments on several activity recognition datasets show that our scheme leads to state-of-the-art performance.