# PAN: Towards Fast Action Recognition via Learning Persistence of Appearance

arXiv: https://arxiv.org/abs/2008.03462 · HTML: https://mlsanity.com/paper/2008.03462/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Aug 2020.

## Built on by (most cited first, 12 of 12)

- [DirecFormer: A Directed Attention in Transformer Approach to Robust Action Recognition](https://mlsanity.com/paper/2203.10233/) (2022)
- [Can An Image Classifier Suffice For Action Recognition?](https://mlsanity.com/paper/2106.14104/) (2021)
- [EAN: Event Adaptive Network for Enhanced Action Recognition](https://mlsanity.com/paper/2107.10771/) (2021)
- [Attention in Attention: Modeling Context Correlation for Efficient Video Classification](https://mlsanity.com/paper/2204.09303/) (2022)
- [Computer Vision for Primate Behavior Analysis in the Wild](https://mlsanity.com/paper/2401.16424/) (2024)
- [Fine-grained Action Analysis: A Multi-modality and Multi-task Dataset of Figure Skating](https://mlsanity.com/paper/2307.02730/) (2023)
- [Explore Spatio-temporal Aggregation for Insubstantial Object Detection: Benchmark Dataset and Baseline](https://mlsanity.com/paper/2206.11459/) (2022)
- [Real-Time Hand Gesture Recognition: Integrating Skeleton-Based Data Fusion and Multi-Stream CNN](https://mlsanity.com/paper/2406.15003/) (2024)
- [DroneAttention: Sparse Weighted Temporal Attention for Drone-Camera Based Activity Recognition](https://mlsanity.com/paper/2212.03384/) (2022)
- [LIGAR: Lightweight General-purpose Action Recognition](https://mlsanity.com/paper/2108.13153/) (2021)
- [Information Elevation Network for Fast Online Action Detection](https://mlsanity.com/paper/2109.13572/) (2021)
- [SWTF: Sparse Weighted Temporal Fusion for Drone-Based Activity Recognition](https://mlsanity.com/paper/2211.05531/) (2022)

## Abstract (arXiv)

Efficiently modeling dynamic motion information in videos is crucial for action recognition task. Most state-of-the-art methods heavily rely on dense optical flow as motion representation. Although combining optical flow with RGB frames as input can achieve excellent recognition performance, the optical flow extraction is very time-consuming. This undoubtably will count against real-time action recognition. In this paper, we shed light on fast action recognition by lifting the reliance on optical flow. Our motivation lies in the observation that small displacements of motion boundaries are the most critical ingredients for distinguishing actions, so we design a novel motion cue called Persistence of Appearance (PA). In contrast to optical flow, our PA focuses more on distilling the motion information at boundaries. Also, it is more efficient by only accumulating pixel-wise differences in feature space, instead of using exhaustive patch-wise search of all the possible motion vectors. Our PA is over 1000x faster (8196fps vs. 8fps) than conventional optical flow in terms of motion modeling speed. To further aggregate the short-term dynamics in PA to long-term dynamics, we also devise a global temporal fusion strategy called Various-timescale Aggregation Pooling (VAP) that can adaptively model long-range temporal relationships across various timescales. We finally incorporate the proposed PA and VAP to form a unified framework called Persistent Appearance Network (PAN) with strong temporal modeling ability. Extensive experiments on six challenging action recognition benchmarks verify that our PAN outperforms recent state-of-the-art methods at low FLOPs. Codes and models are available at: https://github.com/zhang-can/PAN-PyTorch.