# Zero-Shot Action Recognition from Diverse Object-Scene Compositions

arXiv: https://arxiv.org/abs/2110.13479 · HTML: https://mlsanity.com/paper/2110.13479/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 58 references that are in the map.
- First posted on arXiv 26 Oct 2021.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [Video Action Transformer Network](https://mlsanity.com/paper/1812.02707/) (2018)
- [Learning Word Vectors for 157 Languages](https://mlsanity.com/paper/1802.06893/) (2018)
- [Moments in Time Dataset: one million videos for event understanding](https://mlsanity.com/paper/1801.03150/) (2018)
- [Task-Driven Modular Networks for Zero-Shot Compositional Learning](https://mlsanity.com/paper/1905.05908/) (2019)
- [Learning Graph Embeddings for Compositional Zero-shot Learning](https://mlsanity.com/paper/2102.01987/) (2021)
- [Open World Compositional Zero-Shot Learning](https://mlsanity.com/paper/2101.12609/) (2021)
- [A Short Note on the Kinetics-700-2020 Human Action Dataset](https://mlsanity.com/paper/2010.10864/) (2020)
- [Rethinking Zero-shot Video Classification: End-to-end Training for Realistic Applications](https://mlsanity.com/paper/2003.01455/) (2020)
- [Learning Attributes Equals Multi-Source Domain Generalization](https://mlsanity.com/paper/1605.00743/) (2016)
- [Towards Universal Representation for Unseen Action Recognition](https://mlsanity.com/paper/1803.08460/) (2018)
- [Transductive Zero-Shot Action Recognition by Word-Vector Embedding](https://mlsanity.com/paper/1511.04458/) (2015)
- [Objects2action: Classifying and localizing actions without any video example](https://mlsanity.com/paper/1510.06939/) (2015)
- [Semantic Embedding Space for Zero-Shot Action Recognition](https://mlsanity.com/paper/1502.01540/) (2015)
- [Spatial-Aware Object Embeddings for Zero-Shot Localization and Classification of Actions](https://mlsanity.com/paper/1707.09145/) (2017)
- [Object Priors for Classifying and Localizing Unseen Actions](https://mlsanity.com/paper/2104.04715/) (2021)
- [Dynamic Concept Composition for Zero-Example Event Detection](https://mlsanity.com/paper/1601.03679/) (2016)
- [Using Sentences as Semantic Representations in Large Scale Zero-Shot Learning](https://mlsanity.com/paper/2010.02959/) (2020)

## Built on by (most cited first, 8 of 8)

- [Relational Context Learning for Human-Object Interaction Detection](https://mlsanity.com/paper/2304.04997/) (2023)
- [ActionHub: A Large-scale Action Video Description Dataset for Zero-shot Action Recognition](https://mlsanity.com/paper/2401.11654/) (2024)
- [Tell me what you see: A zero-shot action recognition method based on natural language descriptions](https://mlsanity.com/paper/2112.09976/) (2021)
- [Universal Prototype Transport for Zero-Shot Action Recognition and Localization](https://mlsanity.com/paper/2203.03971/) (2022)
- [Global Semantic Descriptors for Zero-Shot Action Recognition](https://mlsanity.com/paper/2209.12061/) (2022)
- [Caption Injection for Optimization in Generative Search Engine](https://mlsanity.com/paper/2511.04080/) (2025)
- [REST: REtrieve & Self-Train for generative action recognition](https://mlsanity.com/paper/2209.15000/) (2022)
- [CEZSAR: A Contrastive Embedding Method for Zero-Shot Action Recognition](https://mlsanity.com/paper/2605.01165/) (2026)

## Abstract (arXiv)

This paper investigates the problem of zero-shot action recognition, in the setting where no training videos with seen actions are available. For this challenging scenario, the current leading approach is to transfer knowledge from the image domain by recognizing objects in videos using pre-trained networks, followed by a semantic matching between objects and actions. Where objects provide a local view on the content in videos, in this work we also seek to include a global view of the scene in which actions occur. We find that scenes on their own are also capable of recognizing unseen actions, albeit more marginally than objects, and a direct combination of object-based and scene-based scores degrades the action recognition performance. To get the best out of objects and scenes, we propose to construct them as a Cartesian product of all possible compositions. We outline how to determine the likelihood of object-scene compositions in videos, as well as a semantic matching from object-scene compositions to actions that enforces diversity among the most relevant compositions for each action. While simple, our composition-based approach outperforms object-based approaches and even state-of-the-art zero-shot approaches that rely on large-scale video datasets with hundreds of seen actions for training and knowledge transfer.