# ActionVLAD: Learning spatio-temporal aggregation for action classification

arXiv: https://arxiv.org/abs/1704.02895 · HTML: https://mlsanity.com/paper/1704.02895/

## Key facts

- Cited by 73 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Apr 2017.

## Built on by (most cited first, 30 of 73)

- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding](https://mlsanity.com/paper/2404.05726/) (2024)
- [Anticipative Video Transformer](https://mlsanity.com/paper/2106.02036/) (2021)
- [MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition](https://mlsanity.com/paper/2201.08383/) (2022)
- [Towards Long-Form Video Understanding](https://mlsanity.com/paper/2106.11310/) (2021)
- [VidTr: Video Transformer Without Convolutions](https://mlsanity.com/paper/2104.11746/) (2021)
- [T2VLAD: Global-Local Sequence Alignment for Text-Video Retrieval](https://mlsanity.com/paper/2104.10054/) (2021)
- [AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?](https://mlsanity.com/paper/2307.16368/) (2023)
- [HierVL: Learning Hierarchical Video-Language Embeddings](https://mlsanity.com/paper/2301.02311/) (2023)
- [SeqNet: Learning Descriptors for Sequence-based Hierarchical Place Recognition](https://mlsanity.com/paper/2102.11603/) (2021)
- [Bridge-Prompt: Towards Ordinal Action Understanding in Instructional Videos](https://mlsanity.com/paper/2203.14104/) (2022)
- [Video Contrastive Learning with Global Context](https://mlsanity.com/paper/2108.02722/) (2021)
- [Anticipative Feature Fusion Transformer for Multi-Modal Action Anticipation](https://mlsanity.com/paper/2210.12649/) (2022)
- [Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphs](https://mlsanity.com/paper/2305.06343/) (2023)
- [Space-Time Crop & Attend: Improving Cross-modal Video Representation Learning](https://mlsanity.com/paper/2103.10211/) (2021)
- [Rethinking Learning Approaches for Long-Term Action Anticipation](https://mlsanity.com/paper/2210.11566/) (2022)
- [PALM: Predicting Actions through Language Models](https://mlsanity.com/paper/2311.17944/) (2023)
- [Coarse to Fine Multi-Resolution Temporal Convolutional Network](https://mlsanity.com/paper/2105.10859/) (2021)
- [No frame left behind: Full Video Action Recognition](https://mlsanity.com/paper/2103.15395/) (2021)
- [C2F-TCN: A Framework for Semi and Fully Supervised Temporal Action Segmentation](https://mlsanity.com/paper/2212.11078/) (2022)
- [Social Fabric: Tubelet Compositions for Video Relation Detection](https://mlsanity.com/paper/2108.08363/) (2021)
- [AutoLabel: CLIP-based framework for Open-set Video Domain Adaptation](https://mlsanity.com/paper/2304.01110/) (2023)
- [TSM: Temporal Shift Module for Efficient and Scalable Video Understanding on Edge Device](https://mlsanity.com/paper/2109.13227/) (2021)
- [Gate-Shift-Fuse for Video Action Recognition](https://mlsanity.com/paper/2203.08897/) (2022)
- [Coarse Temporal Attention Network (CTA-Net) for Driver's Activity Recognition](https://mlsanity.com/paper/2101.06636/) (2021)
- [PromptonomyViT: Multi-Task Prompt Learning Improves Video Transformers using Synthetic Scene Data](https://mlsanity.com/paper/2212.04821/) (2022)
- [DSANet: Dynamic Segment Aggregation Network for Video-Level Representation Learning](https://mlsanity.com/paper/2105.12085/) (2021)
- [Surgical Skill Assessment via Video Semantic Aggregation](https://mlsanity.com/paper/2208.02611/) (2022)
- [CLASTER: Clustering with Reinforcement Learning for Zero-Shot Action Recognition](https://mlsanity.com/paper/2101.07042/) (2021)

## Abstract (arXiv)

In this work, we introduce a new video representation for action classification that aggregates local convolutional features across the entire spatio-temporal extent of the video. We do so by integrating state-of-the-art two-stream networks with learnable spatio-temporal feature aggregation. The resulting architecture is end-to-end trainable for whole-video classification. We investigate different strategies for pooling across space and time and combining signals from the different streams. We find that: (i) it is important to pool jointly across space and time, but (ii) appearance and motion streams are best aggregated into their own separate representations. Finally, we show that our representation outperforms the two-stream base architecture by a large margin (13% relative) as well as out-performs other baselines with comparable base architectures on HMDB51, UCF101, and Charades video classification benchmarks.