# Alternative Semantic Representations for Zero-Shot Human Action Recognition

arXiv: https://arxiv.org/abs/1706.09317 · HTML: https://mlsanity.com/paper/1706.09317/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jun 2017.

## Built on by (most cited first, 27 of 27)

- [Expanding Language-Image Pretrained Models for General Video Recognition](https://mlsanity.com/paper/2208.02816/) (2022)
- [Fine-tuned CLIP Models are Efficient Video Learners](https://mlsanity.com/paper/2212.03640/) (2022)
- [Vita-CLIP: Video and text adaptive CLIP via Multimodal Prompting](https://mlsanity.com/paper/2304.03307/) (2023)
- [Elaborative Rehearsal for Zero-shot Action Recognition](https://mlsanity.com/paper/2108.02833/) (2021)
- [Generative Zero-Shot Learning for Semantic Segmentation of 3D Point Clouds](https://mlsanity.com/paper/2108.06230/) (2021)
- [MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language Knowledge](https://mlsanity.com/paper/2303.08914/) (2023)
- [VicTR: Video-conditioned Text Representations for Activity Recognition](https://mlsanity.com/paper/2304.02560/) (2023)
- [Understanding Long Videos with Multimodal Language Models](https://mlsanity.com/paper/2403.16998/) (2024)
- [OmniVid: A Generative Framework for Universal Video Understanding](https://mlsanity.com/paper/2403.17935/) (2024)
- [Seeing in Flowing: Adapting CLIP for Action Recognition with Motion Prompts Learning](https://mlsanity.com/paper/2308.04828/) (2023)
- [Object Priors for Classifying and Localizing Unseen Actions](https://mlsanity.com/paper/2104.04715/) (2021)
- [Cross-modal Representation Learning for Zero-shot Action Recognition](https://mlsanity.com/paper/2205.01657/) (2022)
- [TVTSv2: Learning Out-of-the-box Spatiotemporal Visual Representations at Scale](https://mlsanity.com/paper/2305.14173/) (2023)
- [EZ-CLIP: Efficient Zeroshot Video Action Recognition](https://mlsanity.com/paper/2312.08010/) (2023)
- [Towards Zero-shot Sign Language Recognition](https://mlsanity.com/paper/2201.05914/) (2022)
- [Video Action Recognition with Attentive Semantic Units](https://mlsanity.com/paper/2303.09756/) (2023)
- [Align before Adapt: Leveraging Entity-to-Region Alignments for Generalizable Video Action Recognition](https://mlsanity.com/paper/2311.15619/) (2023)
- [Language-based Action Concept Spaces Improve Video Self-Supervised Learning](https://mlsanity.com/paper/2307.10922/) (2023)
- [ActionHub: A Large-scale Action Video Description Dataset for Zero-shot Action Recognition](https://mlsanity.com/paper/2401.11654/) (2024)
- [Tell me what you see: A zero-shot action recognition method based on natural language descriptions](https://mlsanity.com/paper/2112.09976/) (2021)
- [Temporally-Adaptive Models for Efficient Video Understanding](https://mlsanity.com/paper/2308.05787/) (2023)
- [Videoprompter: an ensemble of foundational models for zero-shot video understanding](https://mlsanity.com/paper/2310.15324/) (2023)
- [SEZ-HARN: Self-Explainable Zero-shot Human Activity Recognition Network](https://mlsanity.com/paper/2507.00050/) (2025)
- [End-to-End Semantic Video Transformer for Zero-Shot Action Recognition](https://mlsanity.com/paper/2203.05156/) (2022)
- [Learning Using Privileged Information for Zero-Shot Action Recognition](https://mlsanity.com/paper/2206.08632/) (2022)
- [Target inductive methods for zero-shot regression](https://mlsanity.com/paper/2402.01252/) (2024)
- [Direct side information learning for zero-shot regression](https://mlsanity.com/paper/2402.01264/) (2024)

## Abstract (arXiv)

A proper semantic representation for encoding side information is key to the success of zero-shot learning. In this paper, we explore two alternative semantic representations especially for zero-shot human action recognition: textual descriptions of human actions and deep features extracted from still images relevant to human actions. Such side information are accessible on Web with little cost, which paves a new way in gaining side information for large-scale zero-shot human action recognition. We investigate different encoding methods to generate semantic representations for human actions from such side information. Based on our zero-shot visual recognition method, we conducted experiments on UCF101 and HMDB51 to evaluate two proposed semantic representations . The results suggest that our proposed text- and image-based semantic representations outperform traditional attributes and word vectors considerably for zero-shot human action recognition. In particular, the image-based semantic representations yield the favourable performance even though the representation is extracted from a small number of images per class.