# PMI Sampler: Patch Similarity Guided Frame Selection for Aerial Action Recognition

arXiv: https://arxiv.org/abs/2304.06866 · HTML: https://mlsanity.com/paper/2304.06866/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 51 references that are in the map.
- First posted on arXiv 14 Apr 2023.

## Builds on (most cited first, 28 of 28)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [Moments in Time Dataset: one million videos for event understanding](https://mlsanity.com/paper/1801.03150/) (2018)
- [Listen to Look: Action Recognition by Previewing Audio](https://mlsanity.com/paper/1912.04487/) (2019)
- [MoViNets: Mobile Video Networks for Efficient Video Recognition](https://mlsanity.com/paper/2103.11511/) (2021)
- [AdaFrame: Adaptive Frame Selection for Fast Video Recognition](https://mlsanity.com/paper/1811.12432/) (2018)
- [SCSampler: Sampling Salient Clips from Video for Efficient Action Recognition](https://mlsanity.com/paper/1904.04289/) (2019)
- [UAV-Human: A Large Benchmark for Human Behavior Understanding with Unmanned Aerial Vehicles](https://mlsanity.com/paper/2104.00946/) (2021)
- [FrameExit: Conditional Early Exiting for Efficient Video Recognition](https://mlsanity.com/paper/2104.13400/) (2021)
- [MGSampler: An Explainable Sampling Strategy for Video Action Recognition](https://mlsanity.com/paper/2104.09952/) (2021)
- [Okutama-Action: An Aerial View Video Dataset for Concurrent Human Action Detection](https://mlsanity.com/paper/1706.03038/) (2017)
- [One Loss for All: Deep Hashing with a Single Cosine Similarity based Learning Objective](https://mlsanity.com/paper/2109.14449/) (2021)
- [Predicting the Future: A Jointly Learnt Model for Action Anticipation](https://mlsanity.com/paper/1912.07148/) (2019)
- [Dynamic Sampling Networks for Efficient Action Recognition in Videos](https://mlsanity.com/paper/2006.15560/) (2020)
- [OCSampler: Compressing Videos to One Clip with Single-step Sampling](https://mlsanity.com/paper/2201.04388/) (2022)
- [The State of Aerial Surveillance: A Survey](https://mlsanity.com/paper/2201.03080/) (2022)
- [TinyVIRAT: Low-resolution Video Action Recognition](https://mlsanity.com/paper/2007.07355/) (2020)
- [A Multi-viewpoint Outdoor Dataset for Human Action Recognition](https://mlsanity.com/paper/2110.04119/) (2021)
- [FAR: Fourier Aerial Video Recognition](https://mlsanity.com/paper/2203.10694/) (2022)
- [Human Action Recognition in Drone Videos using a Few Aerial Training Examples](https://mlsanity.com/paper/1910.10027/) (2019)
- [AZTR: Aerial Video Action Recognition with Auto Zoom and Temporal Reasoning](https://mlsanity.com/paper/2303.01589/) (2023)
- [Differentiable Frequency-based Disentanglement for Aerial Video Action Recognition](https://mlsanity.com/paper/2209.09194/) (2022)
- [MITFAS: Mutual Information based Temporal Feature Alignment and Sampling for Aerial Video Action Recognition](https://mlsanity.com/paper/2303.02575/) (2023)

## Built on by (most cited first, 7 of 7)

- [Vision-Language Models Learn Super Images for Efficient Partially Relevant Video Retrieval](https://mlsanity.com/paper/2312.00414/) (2023)
- [HawkI: Homography & Mutual Information Guidance for 3D-free Single Image to Aerial View](https://mlsanity.com/paper/2311.15478/) (2023)
- [Beyond the Horizon: Decoupling Multi-View UAV Action Recognition via Partial Order Transfer](https://mlsanity.com/paper/2504.20530/) (2025)
- [SCP: Soft Conditional Prompt Learning for Aerial Video Action Recognition](https://mlsanity.com/paper/2305.12437/) (2023)
- [FALCON: Future-Aware Learning with Contextual Object-Centric Pretraining for UAV Action Recognition](https://mlsanity.com/paper/2409.18300/) (2024)
- [Enhancing Video-Based Robot Failure Detection Using Task Knowledge](https://mlsanity.com/paper/2508.18705/) (2025)
- [Context-Aware Network Based on Multi-scale Spatio-temporal Attention for Action Recognition in Videos](https://mlsanity.com/paper/2512.18750/) (2025)

## Abstract (arXiv)

We present a new algorithm for selection of informative frames in video action recognition. Our approach is designed for aerial videos captured using a moving camera where human actors occupy a small spatial resolution of video frames. Our algorithm utilizes the motion bias within aerial videos, which enables the selection of motion-salient frames. We introduce the concept of patch mutual information (PMI) score to quantify the motion bias between adjacent frames, by measuring the similarity of patches. We use this score to assess the amount of discriminative motion information contained in one frame relative to another. We present an adaptive frame selection strategy using shifted leaky ReLu and cumulative distribution function, which ensures that the sampled frames comprehensively cover all the essential segments with high motion salience. Our approach can be integrated with any action recognition model to enhance its accuracy. In practice, our method achieves a relative improvement of 2.2 - 13.8% in top-1 accuracy on UAV-Human, 6.8% on NEC Drone, and 9.0% on Diving48 datasets.