# Spatial-Aware Object Embeddings for Zero-Shot Localization and Classification of Actions

arXiv: https://arxiv.org/abs/1707.09145 · HTML: https://mlsanity.com/paper/1707.09145/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jul 2017.

## Built on by (most cited first, 19 of 19)

- [High-Dimension Human Value Representation in Large Language Models](https://mlsanity.com/paper/2404.07900/) (2024)
- [Object Priors for Classifying and Localizing Unseen Actions](https://mlsanity.com/paper/2104.04715/) (2021)
- [CLASTER: Clustering with Reinforcement Learning for Zero-Shot Action Recognition](https://mlsanity.com/paper/2101.07042/) (2021)
- [Alignment-Uniformity aware Representation Learning for Zero-shot Video Classification](https://mlsanity.com/paper/2203.15381/) (2022)
- [Zero-Shot Action Recognition from Diverse Object-Scene Compositions](https://mlsanity.com/paper/2110.13479/) (2021)
- [Tell me what you see: A zero-shot action recognition method based on natural language descriptions](https://mlsanity.com/paper/2112.09976/) (2021)
- [Boosting Video Representation Learning with Multi-Faceted Integration](https://mlsanity.com/paper/2201.04023/) (2022)
- [Interaction-Aware Prompting for Zero-Shot Spatio-Temporal Action Detection](https://mlsanity.com/paper/2304.04688/) (2023)
- [Universal Prototype Transport for Zero-Shot Action Recognition and Localization](https://mlsanity.com/paper/2203.03971/) (2022)
- [Telling Stories for Common Sense Zero-Shot Action Recognition](https://mlsanity.com/paper/2309.17327/) (2023)
- [Adaptive Intermediate Representations for Video Understanding](https://mlsanity.com/paper/2104.07135/) (2021)
- [GAN for Vision, KG for Relation: a Two-stage Deep Network for Zero-shot Action Recognition](https://mlsanity.com/paper/2105.11789/) (2021)
- [Global Semantic Descriptors for Zero-Shot Action Recognition](https://mlsanity.com/paper/2209.12061/) (2022)
- [Target inductive methods for zero-shot regression](https://mlsanity.com/paper/2402.01252/) (2024)
- [Few-Shot Transformation of Common Actions into Time and Space](https://mlsanity.com/paper/2104.02439/) (2021)
- [Less than Few: Self-Shot Video Instance Segmentation](https://mlsanity.com/paper/2204.08874/) (2022)
- [Direct side information learning for zero-shot regression](https://mlsanity.com/paper/2402.01264/) (2024)
- [VideoGEM: Training-free Action Grounding in Videos](https://mlsanity.com/paper/2503.20348/) (2025)
- [CEZSAR: A Contrastive Embedding Method for Zero-Shot Action Recognition](https://mlsanity.com/paper/2605.01165/) (2026)

## Abstract (arXiv)

We aim for zero-shot localization and classification of human actions in video. Where traditional approaches rely on global attribute or object classification scores for their zero-shot knowledge transfer, our main contribution is a spatial-aware object embedding. To arrive at spatial awareness, we build our embedding on top of freely available actor and object detectors. Relevance of objects is determined in a word embedding space and further enforced with estimated spatial preferences. Besides local object awareness, we also embed global object awareness into our embedding to maximize actor and object interaction. Finally, we exploit the object positions and sizes in the spatial-aware embedding to demonstrate a new spatio-temporal action retrieval scenario with composite queries. Action localization and classification experiments on four contemporary action video datasets support our proposal. Apart from state-of-the-art results in the zero-shot localization and classification settings, our spatial-aware embedding is even competitive with recent supervised action localization alternatives.