# GTA: Global Temporal Attention for Video Action Understanding

arXiv: https://arxiv.org/abs/2012.08510 · HTML: https://mlsanity.com/paper/2012.08510/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Dec 2020.

## Built on by (most cited first, 20 of 20)

- [Attention Mechanisms in Computer Vision: A Survey](https://mlsanity.com/paper/2111.07624/) (2021)
- [MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding](https://mlsanity.com/paper/2404.05726/) (2024)
- [Align and Attend: Multimodal Summarization with Dual Contrastive Losses](https://mlsanity.com/paper/2303.07284/) (2023)
- [Efficient Video Transformers with Spatial-Temporal Token Selection](https://mlsanity.com/paper/2111.11591/) (2021)
- [ASM-Loc: Action-aware Segment Modeling for Weakly-Supervised Temporal Action Localization](https://mlsanity.com/paper/2203.15187/) (2022)
- [Towards Scalable Neural Representation for Diverse Videos](https://mlsanity.com/paper/2303.14124/) (2023)
- [Learning Semantic Correspondence with Sparse Annotations](https://mlsanity.com/paper/2208.06974/) (2022)
- [Feature Combination Meets Attention: Baidu Soccer Embeddings and Transformer based Temporal Detection](https://mlsanity.com/paper/2106.14447/) (2021)
- [OmniVid: A Generative Framework for Universal Video Understanding](https://mlsanity.com/paper/2403.17935/) (2024)
- [MetaGait: Learning to Learn an Omni Sample Adaptive Representation for Gait Recognition](https://mlsanity.com/paper/2306.03445/) (2023)
- [Chop & Learn: Recognizing and Generating Object-State Compositions](https://mlsanity.com/paper/2309.14339/) (2023)
- [Temporal-attentive Covariance Pooling Networks for Video Recognition](https://mlsanity.com/paper/2110.14381/) (2021)
- [Survey of Action Recognition, Spotting and Spatio-Temporal Localization in Soccer -- Current Trends and Research Perspectives](https://mlsanity.com/paper/2309.12067/) (2023)
- [Deep learning for action spotting in association football videos](https://mlsanity.com/paper/2410.01304/) (2024)
- [Blind VQA on 360deg Video via Progressively Learning from Pixels, Frames and Video](https://mlsanity.com/paper/2111.09503/) (2021)
- [Deep Learning for Sports Video Event Detection: Tasks, Datasets, Methods, and Challenges](https://mlsanity.com/paper/2505.03991/) (2025)
- [M2A: Motion Aware Attention for Accurate Video Action Recognition](https://mlsanity.com/paper/2111.09976/) (2021)
- [Detecting Precise Hand Touch Moments in Egocentric Video](https://mlsanity.com/paper/2604.12343/) (2026)
- [Object-ABN: Learning to Generate Sharp Attention Maps for Action Recognition](https://mlsanity.com/paper/2207.13306/) (2022)
- [PatchBlender: A Motion Prior for Video Transformers](https://mlsanity.com/paper/2211.14449/) (2022)

## Abstract (arXiv)

Self-attention learns pairwise interactions to model long-range dependencies, yielding great improvements for video action recognition. In this paper, we seek a deeper understanding of self-attention for temporal modeling in videos. We first demonstrate that the entangled modeling of spatio-temporal information by flattening all pixels is sub-optimal, failing to capture temporal relationships among frames explicitly. To this end, we introduce Global Temporal Attention (GTA), which performs global temporal attention on top of spatial attention in a decoupled manner. We apply GTA on both pixels and semantically similar regions to capture temporal relationships at different levels of spatial granularity. Unlike conventional self-attention that computes an instance-specific attention matrix, GTA directly learns a global attention matrix that is intended to encode temporal structures that generalize across different samples. We further augment GTA with a cross-channel multi-head fashion to exploit channel interactions for better temporal modeling. Extensive experiments on 2D and 3D networks demonstrate that our approach consistently enhances temporal modeling and provides state-of-the-art performance on three video action recognition datasets.