# A robust and efficient video representation for action recognition

arXiv: https://arxiv.org/abs/1504.05524 · HTML: https://mlsanity.com/paper/1504.05524/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Apr 2015.

## Built on by (most cited first, 20 of 20)

- [Enriching Local and Global Contexts for Temporal Action Localization](https://mlsanity.com/paper/2107.12960/) (2021)
- [MoLo: Motion-augmented Long-short Contrastive Learning for Few-shot Action Recognition](https://mlsanity.com/paper/2304.00946/) (2023)
- [Action Sensitivity Learning for Temporal Action Localization](https://mlsanity.com/paper/2305.15701/) (2023)
- [A Survey on Human-aware Robot Navigation](https://mlsanity.com/paper/2106.11650/) (2021)
- [Hypergraph-based Multi-View Action Recognition using Event Cameras](https://mlsanity.com/paper/2403.19316/) (2024)
- [DailyDVS-200: A Comprehensive Benchmark Dataset for Event-Based Action Recognition](https://mlsanity.com/paper/2407.05106/) (2024)
- [Deep Unsupervised Key Frame Extraction for Efficient Video Classification](https://mlsanity.com/paper/2211.06742/) (2022)
- [Regional Attention with Architecture-Rebuilt 3D Network for RGB-D Gesture Recognition](https://mlsanity.com/paper/2102.05348/) (2021)
- [GestFormer: Multiscale Wavelet Pooling Transformer Network for Dynamic Hand Gesture Recognition](https://mlsanity.com/paper/2405.11180/) (2024)
- [MITFAS: Mutual Information based Temporal Feature Alignment and Sampling for Aerial Video Action Recognition](https://mlsanity.com/paper/2303.02575/) (2023)
- [Can lies be faked? Comparing low-stakes and high-stakes deception video datasets from a Machine Learning perspective](https://mlsanity.com/paper/2211.13035/) (2022)
- [MVTN: A Multiscale Video Transformer Network for Hand Gesture Recognition](https://mlsanity.com/paper/2409.03890/) (2024)
- [ConvMixFormer- A Resource-efficient Convolution Mixer for Transformer-based Dynamic Hand Gesture Recognition](https://mlsanity.com/paper/2411.07118/) (2024)
- [Multiscaled Multi-Head Attention-based Video Transformer Network for Hand Gesture Recognition](https://mlsanity.com/paper/2501.00935/) (2025)
- [Learning Using Privileged Information for Zero-Shot Action Recognition](https://mlsanity.com/paper/2206.08632/) (2022)
- [PoseAction: Action Recognition for Patients in the Ward using Deep Learning Approaches](https://mlsanity.com/paper/2310.03288/) (2023)
- [Action parsing using context features](https://mlsanity.com/paper/2205.10008/) (2022)
- [Online Hand Gesture Recognition Using 3D Convolutional Neural Networks](https://mlsanity.com/paper/2605.23409/) (2026)
- [City Sentinel: A Unified AI-Based Smart Surveillance Framework for Real-Time Multi-Threat Detection Using Deep Learning](https://mlsanity.com/paper/2608.08887/) (2026)
- [UniCon-Former: Unified Convolution Transformer is All You Need for Hand Gesture Recognition](https://mlsanity.com/paper/2608.13217/) (2026)

## Abstract (arXiv)

This paper introduces a state-of-the-art video representation and applies it to efficient action recognition and detection. We first propose to improve the popular dense trajectory features by explicit camera motion estimation. More specifically, we extract feature point matches between frames using SURF descriptors and dense optical flow. The matches are used to estimate a homography with RANSAC. To improve the robustness of homography estimation, a human detector is employed to remove outlier matches from the human body as human motion is not constrained by the camera. Trajectories consistent with the homography are considered as due to camera motion, and thus removed. We also use the homography to cancel out camera motion from the optical flow. This results in significant improvement on motion-based HOF and MBH descriptors. We further explore the recent Fisher vector as an alternative feature encoding approach to the standard bag-of-words histogram, and consider different ways to include spatial layout information in these encodings. We present a large and varied set of evaluations, considering (i) classification of short basic actions on six datasets, (ii) localization of such actions in feature-length movies, and (iii) large-scale recognition of complex events. We find that our improved trajectory features significantly outperform previous dense trajectories, and that Fisher vectors are superior to bag-of-words encodings for video recognition tasks. In all three tasks, we show substantial improvements over the state-of-the-art results.