# Motion-Guided Semantic Alignment with Negative Prompts for Zero-Shot Video Action Recognition

arXiv: https://arxiv.org/abs/2604.17062 · HTML: https://mlsanity.com/paper/2604.17062/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 26 references that are in the map.
- First posted on arXiv 18 Apr 2026.

## Builds on (most cited first, 21 of 21)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [A Short Note about Kinetics-600](https://mlsanity.com/paper/1808.01340/) (2018)
- [Expanding Language-Image Pretrained Models for General Video Recognition](https://mlsanity.com/paper/2208.02816/) (2022)
- [Prompting Visual-Language Models for Efficient Video Understanding](https://mlsanity.com/paper/2112.04478/) (2021)
- [Fine-tuned CLIP Models are Efficient Video Learners](https://mlsanity.com/paper/2212.03640/) (2022)
- [Vita-CLIP: Video and text adaptive CLIP via Multimodal Prompting](https://mlsanity.com/paper/2304.03307/) (2023)
- [Elaborative Rehearsal for Zero-shot Action Recognition](https://mlsanity.com/paper/2108.02833/) (2021)
- [Understanding the Impact of Negative Prompts: When and How Do They Take Effect?](https://mlsanity.com/paper/2406.02965/) (2024)
- [MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language Knowledge](https://mlsanity.com/paper/2303.08914/) (2023)
- [FROSTER: Frozen CLIP Is A Strong Teacher for Open-Vocabulary Action Recognition](https://mlsanity.com/paper/2402.03241/) (2024)
- [Deep Adaptive Wavelet Network](https://mlsanity.com/paper/1912.05035/) (2019)
- [EZ-CLIP: Efficient Zeroshot Video Action Recognition](https://mlsanity.com/paper/2312.08010/) (2023)
- [Building a Multi-modal Spatiotemporal Expert for Zero-shot Action Recognition with CLIP](https://mlsanity.com/paper/2412.09895/) (2024)
- [JigsawNet: Shredded Image Reassembly using Convolutional Neural Network and Loop-based Composition](https://mlsanity.com/paper/1809.04137/) (2018)
- [Continual Learning Improves Zero-Shot Action Recognition](https://mlsanity.com/paper/2410.10497/) (2024)
- [Telling Stories for Common Sense Zero-Shot Action Recognition](https://mlsanity.com/paper/2309.17327/) (2023)
- [Kronecker Mask and Interpretive Prompts are Language-Action Video Learners](https://mlsanity.com/paper/2502.03549/) (2025)
- [Is Temporal Prompting All We Need For Limited Labeled Action Recognition?](https://mlsanity.com/paper/2504.01890/) (2025)

## Abstract (arXiv)

Zero-shot action recognition is challenging due to the semantic gap between seen and unseen classes. We present a novel framework that enhances CLIP with disentangled embeddings and semantic-guided interaction. A Motion Separation Module (MSM) separates motion-sensitive and global-static features, while a Motion Aggregation Block (MAB) employs gated cross-attention to refine motion representation without re-coupling redundant information. To facilitate generalization to unseen categories, we enforce semantic alignment between video features and textual representations by aligning projected embeddings with positive textual prompts, while leveraging negative prompts to explicitly model "non-class" semantics. Experiments on standard benchmarks demonstrate that our method consistently outperforms prior CLIP-based approaches, achieving robust zero-shot action recognition across both coarse and fine-grained datasets.