# The Wisdom of Crowds: Temporal Progressive Attention for Early Action Prediction

arXiv: https://arxiv.org/abs/2204.13340 · HTML: https://mlsanity.com/paper/2204.13340/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 76 references that are in the map.
- First posted on arXiv 28 Apr 2022.

## Builds on (most cited first, 30 of 42)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020)
- [A Short Note about Kinetics-600](https://mlsanity.com/paper/1808.01340/) (2018)
- [Temporal Relational Reasoning in Videos](https://mlsanity.com/paper/1711.08496/) (2017)
- [Long-Term Feature Banks for Detailed Video Understanding](https://mlsanity.com/paper/1812.05038/) (2018)
- [Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding](https://mlsanity.com/paper/2103.15358/) (2021)
- [Anticipative Video Transformer](https://mlsanity.com/paper/2106.02036/) (2021)
- [MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition](https://mlsanity.com/paper/2201.08383/) (2022)
- [Multiview Transformers for Video Recognition](https://mlsanity.com/paper/2201.04288/) (2022)
- [Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolution](https://mlsanity.com/paper/1904.05049/) (2019)
- [MoViNets: Mobile Video Networks for Efficient Video Recognition](https://mlsanity.com/paper/2103.11511/) (2021)
- [AdaFrame: Adaptive Frame Selection for Fast Video Recognition](https://mlsanity.com/paper/1811.12432/) (2018)

## Built on by (most cited first, 13 of 13)

- [CaptainCook4D: A Dataset for Understanding Errors in Procedural Activities](https://mlsanity.com/paper/2312.14556/) (2023)
- [Fostering Video Reasoning via Next-Event Prediction](https://mlsanity.com/paper/2505.22457/) (2025)
- [GenRec: Unifying Video Generation and Recognition with Diffusion Models](https://mlsanity.com/paper/2408.15241/) (2024)
- [Online Episodic Memory Visual Query Localization with Egocentric Streaming Object Memory](https://mlsanity.com/paper/2411.16934/) (2024)
- [MuST: Multi-Scale Transformers for Surgical Phase Recognition](https://mlsanity.com/paper/2407.17361/) (2024)
- [Uni-Hand: Universal Hand Motion Forecasting in Egocentric Views](https://mlsanity.com/paper/2511.12878/) (2025)
- [MistExit: Learning to Exit for Early Mistake Detection in Procedural Videos](https://mlsanity.com/paper/2603.14252/) (2026)
- [Early Action Recognition with Action Prototypes](https://mlsanity.com/paper/2312.06598/) (2023)
- [Early Operative Difficulty Assessment in Laparoscopic Cholecystectomy via Snapshot-Centric Video Analysis](https://mlsanity.com/paper/2502.07008/) (2025)
- [Beyond Visual CoT: Internalized Visual Thinking for Proactive Video Reasoning](https://mlsanity.com/paper/2608.15869/) (2026)
- [EAST: Early Action Prediction Sampling Strategy with Token Masking](https://mlsanity.com/paper/2604.18367/) (2026)
- [High-Speed Vision Improves Zero-Shot Semantic Understanding of Human Actions](https://mlsanity.com/paper/2605.00496/) (2026)
- [Zero-Shot Skeleton-Based Action Anticipation](https://mlsanity.com/paper/2608.14243/) (2026)

## Abstract (arXiv)

Early action prediction deals with inferring the ongoing action from partially-observed videos, typically at the outset of the video. We propose a bottleneck-based attention model that captures the evolution of the action, through progressive sampling over fine-to-coarse scales. Our proposed Temporal Progressive (TemPr) model is composed of multiple attention towers, one for each scale. The predicted action label is based on the collective agreement considering confidences of these towers. Extensive experiments over four video datasets showcase state-of-the-art performance on the task of Early Action Prediction across a range of encoder architectures. We demonstrate the effectiveness and consistency of TemPr through detailed ablations.