# Objects2action: Classifying and localizing actions without any video example

arXiv: https://arxiv.org/abs/1510.06939 · HTML: https://mlsanity.com/paper/1510.06939/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Oct 2015.

## Built on by (most cited first, 26 of 26)

- [Prompting Visual-Language Models for Efficient Video Understanding](https://mlsanity.com/paper/2112.04478/) (2021)
- [Elaborative Rehearsal for Zero-shot Action Recognition](https://mlsanity.com/paper/2108.02833/) (2021)
- [Open-VCLIP: Transforming CLIP to an Open-vocabulary Video Model via Interpolated Weight Optimization](https://mlsanity.com/paper/2302.00624/) (2023)
- [Zero-shot Natural Language Video Localization](https://mlsanity.com/paper/2110.00428/) (2021)
- [Pseudo-Q: Generating Pseudo Language Queries for Visual Grounding](https://mlsanity.com/paper/2203.08481/) (2022)
- [Building an Open-Vocabulary Video CLIP Model with Better Architectures, Optimization and Data](https://mlsanity.com/paper/2310.05010/) (2023)
- [FitCLIP: Refining Large-Scale Pretrained Image-Text Models for Zero-Shot Video Understanding Tasks](https://mlsanity.com/paper/2203.13371/) (2022)
- [Object Priors for Classifying and Localizing Unseen Actions](https://mlsanity.com/paper/2104.04715/) (2021)
- [XOV-Action: Towards Generalizable Open-Vocabulary Action Recognition](https://mlsanity.com/paper/2403.01560/) (2024)
- [Towards Zero-shot Sign Language Recognition](https://mlsanity.com/paper/2201.05914/) (2022)
- [Alignment-Uniformity aware Representation Learning for Zero-shot Video Classification](https://mlsanity.com/paper/2203.15381/) (2022)
- [Language-based Action Concept Spaces Improve Video Self-Supervised Learning](https://mlsanity.com/paper/2307.10922/) (2023)
- [Building a Multi-modal Spatiotemporal Expert for Zero-shot Action Recognition with CLIP](https://mlsanity.com/paper/2412.09895/) (2024)
- [Zero-Shot Action Recognition from Diverse Object-Scene Compositions](https://mlsanity.com/paper/2110.13479/) (2021)
- [ActionHub: A Large-scale Action Video Description Dataset for Zero-shot Action Recognition](https://mlsanity.com/paper/2401.11654/) (2024)
- [Tell me what you see: A zero-shot action recognition method based on natural language descriptions](https://mlsanity.com/paper/2112.09976/) (2021)
- [Interaction-Aware Prompting for Zero-Shot Spatio-Temporal Action Detection](https://mlsanity.com/paper/2304.04688/) (2023)
- [Universal Prototype Transport for Zero-Shot Action Recognition and Localization](https://mlsanity.com/paper/2203.03971/) (2022)
- [GAN for Vision, KG for Relation: a Two-stage Deep Network for Zero-shot Action Recognition](https://mlsanity.com/paper/2105.11789/) (2021)
- [Global Semantic Descriptors for Zero-Shot Action Recognition](https://mlsanity.com/paper/2209.12061/) (2022)
- [TNT: Text-Conditioned Network with Transductive Inference for Few-Shot Video Classification](https://mlsanity.com/paper/2106.11173/) (2021)
- [Learning Using Privileged Information for Zero-Shot Action Recognition](https://mlsanity.com/paper/2206.08632/) (2022)
- [ACE: Action Concept Enhancement of Video-Language Models in Procedural Videos](https://mlsanity.com/paper/2411.15628/) (2024)
- [Few-Shot Transformation of Common Actions into Time and Space](https://mlsanity.com/paper/2104.02439/) (2021)
- [How Do You Do It? Fine-Grained Action Understanding with Pseudo-Adverbs](https://mlsanity.com/paper/2203.12344/) (2022)
- [Less than Few: Self-Shot Video Instance Segmentation](https://mlsanity.com/paper/2204.08874/) (2022)

## Abstract (arXiv)

The goal of this paper is to recognize actions in video without the need for examples. Different from traditional zero-shot approaches we do not demand the design and specification of attribute classifiers and class-to-attribute mappings to allow for transfer from seen classes to unseen classes. Our key contribution is objects2action, a semantic word embedding that is spanned by a skip-gram model of thousands of object categories. Action labels are assigned to an object encoding of unseen video based on a convex combination of action and object affinities. Our semantic embedding has three main characteristics to accommodate for the specifics of actions. First, we propose a mechanism to exploit multiple-word descriptions of actions and objects. Second, we incorporate the automated selection of the most responsive objects per action. And finally, we demonstrate how to extend our zero-shot approach to the spatio-temporal localization of actions in video. Experiments on four action datasets demonstrate the potential of our approach.