# Capturing Temporal Information in a Single Frame: Channel Sampling Strategies for Action Recognition

arXiv: https://arxiv.org/abs/2201.10394 · HTML: https://mlsanity.com/paper/2201.10394/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 28 references that are in the map.
- First posted on arXiv 25 Jan 2022.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [A simple neural network module for relational reasoning](https://mlsanity.com/paper/1706.01427/) (2017)
- [Temporal Relational Reasoning in Videos](https://mlsanity.com/paper/1711.08496/) (2017)
- [Temporal Segment Networks for Action Recognition in Videos](https://mlsanity.com/paper/1705.02953/) (2017)
- [TDN: Temporal Difference Networks for Efficient Action Recognition](https://mlsanity.com/paper/2012.10071/) (2020)
- [CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning](https://mlsanity.com/paper/1910.04744/) (2019)
- [Compressed Video Action Recognition](https://mlsanity.com/paper/1712.00636/) (2017)
- [SMART Frame Selection for Action Recognition](https://mlsanity.com/paper/2012.10671/) (2020)
- [Only Time Can Tell: Discovering Temporal Data for Temporal Modeling](https://mlsanity.com/paper/1907.08340/) (2019)
- [Real-time Action Recognition with Enhanced Motion Vector CNNs](https://mlsanity.com/paper/1604.07669/) (2016)
- [MVFNet: Multi-View Fusion Network for Efficient Video Recognition](https://mlsanity.com/paper/2012.06977/) (2020)

## Built on by (most cited first, 10 of 10)

- [Text-Conditioned Resampler For Long Form Video Understanding](https://mlsanity.com/paper/2312.11897/) (2023)
- [Watt For What: Rethinking Deep Learning's Energy-Performance Relationship](https://mlsanity.com/paper/2310.06522/) (2023)
- [Taylor Videos for Action Recognition](https://mlsanity.com/paper/2402.03019/) (2024)
- [Learning a Condensed Frame for Memory-Efficient Video Class-Incremental Learning](https://mlsanity.com/paper/2211.00833/) (2022)
- [Benchmarks for Physical Reasoning AI](https://mlsanity.com/paper/2312.10728/) (2023)
- [PATS: Proficiency-Aware Temporal Sampling for Multi-View Sports Skill Assessment](https://mlsanity.com/paper/2506.04996/) (2025)
- [Telling Stories for Common Sense Zero-Shot Action Recognition](https://mlsanity.com/paper/2309.17327/) (2023)
- [When Spatial meets Temporal in Action Recognition](https://mlsanity.com/paper/2411.15284/) (2024)
- [Adversarial Augmentation Training Makes Action Recognition Models More Robust to Realistic Video Distribution Shifts](https://mlsanity.com/paper/2401.11406/) (2024)
- [Novel Semantic Prompting for Zero-Shot Action Recognition](https://mlsanity.com/paper/2603.08289/) (2026)

## Abstract (arXiv)

We address the problem of capturing temporal information for video classification in 2D networks, without increasing their computational cost. Existing approaches focus on modifying the architecture of 2D networks (e.g. by including filters in the temporal dimension to turn them into 3D networks, or using optical flow, etc.), which increases computation cost. Instead, we propose a novel sampling strategy, where we re-order the channels of the input video, to capture short-term frame-to-frame changes. We observe that without bells and whistles, the proposed sampling strategy improves performance on multiple architectures (e.g. TSN, TRN, TSM, and MVFNet) and datasets (CATER, Something-Something-V1 and V2), up to 24% over the baseline of using the standard video input. In addition, our sampling strategies do not require training from scratch and do not increase the computational cost of training and testing. Given the generality of the results and the flexibility of the approach, we hope this can be widely useful to the video understanding community. Code is available on our website: https://github.com/kiyoon/channel_sampling.