# Elaborative Rehearsal for Zero-shot Action Recognition

arXiv: https://arxiv.org/abs/2108.02833 · HTML: https://mlsanity.com/paper/2108.02833/

## Key facts

- Cited by 77 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 61 references that are in the map.
- First posted on arXiv 5 Aug 2021.

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?](https://mlsanity.com/paper/2003.11539/) (2020)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [A Short Note about Kinetics-600](https://mlsanity.com/paper/1808.01340/) (2018)
- [Temporal Segment Networks for Action Recognition in Videos](https://mlsanity.com/paper/1705.02953/) (2017)
- [Evaluation of Output Embeddings for Fine-Grained Image Classification](https://mlsanity.com/paper/1409.8403/) (2014)
- [Label-Embedding for Image Classification](https://mlsanity.com/paper/1503.08677/) (2015)
- [Zero-shot Recognition via Semantic Embeddings and Knowledge Graphs](https://mlsanity.com/paper/1803.08035/) (2018)
- [Learning a Deep Embedding Model for Zero-Shot Learning](https://mlsanity.com/paper/1611.05088/) (2016)
- [Large-scale weakly-supervised pre-training for video action recognition](https://mlsanity.com/paper/1905.00561/) (2019)
- [Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features](https://mlsanity.com/paper/1703.02507/) (2017)
- [Rethinking Zero-shot Video Classification: End-to-end Training for Realistic Applications](https://mlsanity.com/paper/2003.01455/) (2020)
- [Out-of-Distribution Detection for Generalized Zero-Shot Action Recognition](https://mlsanity.com/paper/1904.08703/) (2019)
- [Learning Attributes Equals Multi-Source Domain Generalization](https://mlsanity.com/paper/1605.00743/) (2016)
- [Towards Universal Representation for Unseen Action Recognition](https://mlsanity.com/paper/1803.08460/) (2018)
- [Less is more: zero-shot learning from online textual documents with noise suppression](https://mlsanity.com/paper/1604.01146/) (2016)
- [Transductive Zero-Shot Action Recognition by Word-Vector Embedding](https://mlsanity.com/paper/1511.04458/) (2015)
- [Multi-Task Zero-Shot Action Recognition with Prioritised Data Augmentation](https://mlsanity.com/paper/1611.08663/) (2016)
- [Alternative Semantic Representations for Zero-Shot Human Action Recognition](https://mlsanity.com/paper/1706.09317/) (2017)
- [Objects2action: Classifying and localizing actions without any video example](https://mlsanity.com/paper/1510.06939/) (2015)

## Built on by (most cited first, 30 of 77)

- [Expanding Language-Image Pretrained Models for General Video Recognition](https://mlsanity.com/paper/2208.02816/) (2022)
- [Fine-tuned CLIP Models are Efficient Video Learners](https://mlsanity.com/paper/2212.03640/) (2022)
- [VideoPrism: A Foundational Visual Encoder for Video Understanding](https://mlsanity.com/paper/2402.13217/) (2024)
- [Vita-CLIP: Video and text adaptive CLIP via Multimodal Prompting](https://mlsanity.com/paper/2304.03307/) (2023)
- [Verbs in Action: Improving verb understanding in video-language models](https://mlsanity.com/paper/2304.06708/) (2023)
- [Revisiting Classifier: Transferring Vision-Language Models for Video Recognition](https://mlsanity.com/paper/2207.01297/) (2022)
- [VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captioners](https://mlsanity.com/paper/2212.04979/) (2022)
- [Open-VCLIP: Transforming CLIP to an Open-vocabulary Video Model via Interpolated Weight Optimization](https://mlsanity.com/paper/2302.00624/) (2023)
- [Bidirectional Cross-Modal Knowledge Exploration for Video Recognition with Pre-trained Vision-Language Models](https://mlsanity.com/paper/2301.00182/) (2022)
- [MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language Knowledge](https://mlsanity.com/paper/2303.08914/) (2023)
- [FROSTER: Frozen CLIP Is A Strong Teacher for Open-Vocabulary Action Recognition](https://mlsanity.com/paper/2402.03241/) (2024)
- [Pseudo-Q: Generating Pseudo Language Queries for Visual Grounding](https://mlsanity.com/paper/2203.08481/) (2022)
- [VicTR: Video-conditioned Text Representations for Activity Recognition](https://mlsanity.com/paper/2304.02560/) (2023)
- [AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportation](https://mlsanity.com/paper/2407.04603/) (2024)
- [TENT: Connect Language Models with IoT Sensors for Zero-Shot Activity Recognition](https://mlsanity.com/paper/2311.08245/) (2023)
- [Multimodal Open-Vocabulary Video Classification via Pre-Trained Vision and Language Models](https://mlsanity.com/paper/2207.07646/) (2022)
- [Building an Open-Vocabulary Video CLIP Model with Better Architectures, Optimization and Data](https://mlsanity.com/paper/2310.05010/) (2023)
- [EdgeFM: Leveraging Foundation Model for Open-set Learning on the Edge](https://mlsanity.com/paper/2311.10986/) (2023)
- [From Isolated Islands to Pangea: Unifying Semantic Space for Human Action Understanding](https://mlsanity.com/paper/2304.00553/) (2023)
- [FitCLIP: Refining Large-Scale Pretrained Image-Text Models for Zero-Shot Video Understanding Tasks](https://mlsanity.com/paper/2203.13371/) (2022)
- [Seeing in Flowing: Adapting CLIP for Action Recognition with Motion Prompts Learning](https://mlsanity.com/paper/2308.04828/) (2023)
- [OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognition](https://mlsanity.com/paper/2312.00096/) (2023)
- [Cross-modal Representation Learning for Zero-shot Action Recognition](https://mlsanity.com/paper/2205.01657/) (2022)
- [XOV-Action: Towards Generalizable Open-Vocabulary Action Recognition](https://mlsanity.com/paper/2403.01560/) (2024)
- [TVTSv2: Learning Out-of-the-box Spatiotemporal Visual Representations at Scale](https://mlsanity.com/paper/2305.14173/) (2023)
- [EZ-CLIP: Efficient Zeroshot Video Action Recognition](https://mlsanity.com/paper/2312.08010/) (2023)
- [OmniCLIP: Adapting CLIP for Video Recognition with Spatial-Temporal Omni-Scale Feature Learning](https://mlsanity.com/paper/2408.06158/) (2024)
- [Video Action Recognition with Attentive Semantic Units](https://mlsanity.com/paper/2303.09756/) (2023)
- [Align before Adapt: Leveraging Entity-to-Region Alignments for Generalizable Video Action Recognition](https://mlsanity.com/paper/2311.15619/) (2023)
- [Fine-Grained Zero-Shot Learning: Advances, Challenges, and Prospects](https://mlsanity.com/paper/2401.17766/) (2024)

## Abstract (arXiv)

The growing number of action classes has posed a new challenge for video understanding, making Zero-Shot Action Recognition (ZSAR) a thriving direction. The ZSAR task aims to recognize target (unseen) actions without training examples by leveraging semantic representations to bridge seen and unseen actions. However, due to the complexity and diversity of actions, it remains challenging to semantically represent action classes and transfer knowledge from seen data. In this work, we propose an ER-enhanced ZSAR model inspired by an effective human memory technique Elaborative Rehearsal (ER), which involves elaborating a new concept and relating it to known concepts. Specifically, we expand each action class as an Elaborative Description (ED) sentence, which is more discriminative than a class name and less costly than manual-defined attributes. Besides directly aligning class semantics with videos, we incorporate objects from the video as Elaborative Concepts (EC) to improve video semantics and generalization from seen actions to unseen actions. Our ER-enhanced ZSAR model achieves state-of-the-art results on three existing benchmarks. Moreover, we propose a new ZSAR evaluation protocol on the Kinetics dataset to overcome limitations of current benchmarks and demonstrate the first case where ZSAR performance is comparable to few-shot learning baselines on this more realistic setting. We will release our codes and collected EDs at https://github.com/DeLightCMU/ElaborativeRehearsal.