# Enhancing Spatio-Temporal Zero-shot Action Recognition with Language-driven Description Attributes

arXiv: https://arxiv.org/abs/2510.27255 · HTML: https://mlsanity.com/paper/2510.27255/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 21 of its 37 references that are in the map.
- First posted on arXiv 31 Oct 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Builds on (most cited first, 21 of 21)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [A Short Note about Kinetics-600](https://mlsanity.com/paper/1808.01340/) (2018)
- [Expanding Language-Image Pretrained Models for General Video Recognition](https://mlsanity.com/paper/2208.02816/) (2022)
- [ActionCLIP: A New Paradigm for Video Action Recognition](https://mlsanity.com/paper/2109.08472/) (2021)
- [Fine-tuned CLIP Models are Efficient Video Learners](https://mlsanity.com/paper/2212.03640/) (2022)
- [Multiview Transformers for Video Recognition](https://mlsanity.com/paper/2201.04288/) (2022)
- [Elaborative Rehearsal for Zero-shot Action Recognition](https://mlsanity.com/paper/2108.02833/) (2021)
- [Revisiting Classifier: Transferring Vision-Language Models for Video Recognition](https://mlsanity.com/paper/2207.01297/) (2022)
- [Bidirectional Cross-Modal Knowledge Exploration for Video Recognition with Pre-trained Vision-Language Models](https://mlsanity.com/paper/2301.00182/) (2022)
- [MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language Knowledge](https://mlsanity.com/paper/2303.08914/) (2023)
- [Co-training Transformer with Videos and Images Improves Action Recognition](https://mlsanity.com/paper/2112.07175/) (2021)
- [GPT4Vis: What Can GPT-4 Do for Zero-shot Visual Recognition?](https://mlsanity.com/paper/2311.15732/) (2023)
- [VicTR: Video-conditioned Text Representations for Activity Recognition](https://mlsanity.com/paper/2304.02560/) (2023)
- [Auxiliary Tasks Benefit 3D Skeleton-based Human Motion Prediction](https://mlsanity.com/paper/2308.08942/) (2023)
- [Video Action Recognition with Attentive Semantic Units](https://mlsanity.com/paper/2303.09756/) (2023)
- [Videoprompter: an ensemble of foundational models for zero-shot video understanding](https://mlsanity.com/paper/2310.15324/) (2023)
- [Spatio-Temporal Context Prompting for Zero-Shot Action Detection](https://mlsanity.com/paper/2408.15996/) (2024)

## Built on by (most cited first, 1 of 1)

- [Thinking Before Retrieving: Robust Zero-Shot Composed Image Retrieval via Strategic Planning and Self-Criticism](https://mlsanity.com/paper/2606.31222/) (2026)

## Abstract (arXiv)

Vision-Language Models (VLMs) have demonstrated impressive capabilities in zero-shot action recognition by learning to associate video embeddings with class embeddings. However, a significant challenge arises when relying solely on action classes to provide semantic context, particularly due to the presence of multi-semantic words, which can introduce ambiguity in understanding the intended concepts of actions. To address this issue, we propose an innovative approach that harnesses web-crawled descriptions, leveraging a large-language model to extract relevant keywords. This method reduces the need for human annotators and eliminates the laborious manual process of attribute data creation. Additionally, we introduce a spatio-temporal interaction module designed to focus on objects and action units, facilitating alignment between description attributes and video content. In our zero-shot experiments, our model achieves impressive results, attaining accuracies of 81.0%, 53.1%, and 68.9% on UCF-101, HMDB-51, and Kinetics-600, respectively, underscoring the model's adaptability and effectiveness across various downstream tasks.