mlsanityData to 7 Oct 2026

Novel Semantic Prompting for Zero-Shot Action Recognition

arXiv 2603.08289 ↗First posted 9 Mar 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Zero-shot action recognition relies on transferring knowledge from vision-language models to unseen actions using semantic descriptions. While recent methods focus on temporal modeling or architectural adaptations to handle video data, we argue that semantic prompting alone provides a strong and underexplored signal for zero-shot action understanding. We introduce SP-CLIP, a lightweight framework that augments frozen vision-language models with structured semantic prompts describing actions at multiple levels of abstraction, such as intent, motion, and object interaction. Without modifying the visual encoder or learning additional parameters, SP-CLIP aligns video representations with enriched textual semantics through prompt aggregation and consistency scoring. Experiments across standard benchmarks show that semantic prompting substantially improves zero-shot action recognition, particularly for fine-grained and compositional actions, while preserving the efficiency and generalization of pretrained models.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 33 references that are in the map. First posted on arXiv 9 Mar 2026.

Builds on

19 of 33 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Is Temporal Prompting All We Need For Limited Labeled Action Recognition?2 Apr

Other references · 18

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  4. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  5. 2017Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Uglycited by 536
  6. 2020Rethinking Zero-shot Video Classification: End-to-end Training for Realistic Applicationscited by 79
  7. 2021Elaborative Rehearsal for Zero-shot Action Recognitioncited by 76
  8. 2019Out-of-Distribution Detection for Generalized Zero-Shot Action Recognitioncited by 54
  9. 2018A Generative Approach to Zero-Shot and Few-Shot Action Recognitioncited by 39
  10. 2019Collaborative Spatio-temporal Feature Learning for Video Action Recognitioncited by 24
  11. 2015Semantic Embedding Space for Zero-Shot Action Recognitioncited by 19
  12. 2021A New Split for Evaluating True Zero-Shot Action Recognitioncited by 14
6 more
  1. 2023Synthetic Sample Selection for Generalized Zero-Shot Learningcited by 14
  2. 2023EZ-CLIP: Efficient Zeroshot Video Action Recognitioncited by 14
  3. 2021CLASTER: Clustering with Reinforcement Learning for Zero-Shot Action Recognitioncited by 12
  4. 2022Capturing Temporal Information in a Single Frame: Channel Sampling Strategies for Action Recognitioncited by 10
  5. 2024Continual Learning Improves Zero-Shot Action Recognitioncited by 5
  6. 2023Telling Stories for Common Sense Zero-Shot Action Recognitioncited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Knowledge Prompting for Few-shot Action Recognition
  2. 2025Enhancing Spatio-Temporal Zero-shot Action Recognition with Language-driven Description Attributes
  3. 2023ZEETAD: Adapting Pretrained Vision-Language Model for Zero-Shot End-to-End Temporal Action Detection
  4. 2026Exploring Vision-Language Models for Open-Vocabulary Zero-Shot Action Segmentation
  5. 2026High-Speed Vision Improves Zero-Shot Semantic Understanding of Human Actions
  6. 2026EV-CLIP: Efficient Visual Prompt Adaptation for CLIP in Few-shot Action Recognition under Visual Challenges
  7. 2022Multi-Modal Few-Shot Temporal Action Detection
  8. 2024Text-Enhanced Zero-Shot Action Recognition: A training-free approach