# Fine-Grained Side Information Guided Dual-Prompts for Zero-Shot Skeleton Action Recognition

arXiv: https://arxiv.org/abs/2404.07487 · HTML: https://mlsanity.com/paper/2404.07487/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 79 references that are in the map.
- First posted on arXiv 11 Apr 2024.

## Builds on (most cited first, 30 of 53)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2107.12213/) (2021)
- [Deep Learning-Based Human Pose Estimation: A Survey](https://mlsanity.com/paper/2012.13392/) (2020)
- [A Review of Generalized Zero-Shot Learning Methods](https://mlsanity.com/paper/2011.08641/) (2020)
- [An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wild](https://mlsanity.com/paper/1605.04253/) (2016)
- [PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding](https://mlsanity.com/paper/1703.07475/) (2017)
- [Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings](https://mlsanity.com/paper/1908.03477/) (2019)
- [Semantic-Guided Multi-Attention Localization for Zero-Shot Learning](https://mlsanity.com/paper/1903.00502/) (2019)
- [Learning Robust Visual-Semantic Embeddings](https://mlsanity.com/paper/1703.05908/) (2017)
- [TransZero: Attribute-guided Transformer for Zero-Shot Learning](https://mlsanity.com/paper/2112.01683/) (2021)
- [Decomposed Soft Prompt Guided Fusion Enhancing for Compositional Zero-Shot Learning](https://mlsanity.com/paper/2211.10681/) (2022)
- [DiPrompT: Disentangled Prompt Tuning for Multiple Latent Domain Generalization in Federated Learning](https://mlsanity.com/paper/2403.08506/) (2024)
- [A Novel Perspective to Zero-shot Learning: Towards an Alignment of Manifold Structures via Semantic Feature Expansion](https://mlsanity.com/paper/2004.14795/) (2020)
- [TransZero++: Cross Attribute-Guided Transformer for Zero-Shot Learning](https://mlsanity.com/paper/2112.08643/) (2021)
- [SRCD: Semantic Reasoning with Compound Domains for Single-Domain Generalized Object Detection](https://mlsanity.com/paper/2307.01750/) (2023)
- [I2DFormer: Learning Image to Document Attention for Zero-Shot Image Classification](https://mlsanity.com/paper/2209.10304/) (2022)
- [Discriminative Learning of Latent Features for Zero-Shot Recognition](https://mlsanity.com/paper/1803.06731/) (2018)
- [Syntactically Guided Generative Embeddings for Zero-Shot Skeleton Action Recognition](https://mlsanity.com/paper/2101.11530/) (2021)
- [Progressive Semantic-Visual Mutual Adaption for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2303.15322/) (2023)
- [Zero-shot Skeleton-based Action Recognition via Mutual Information Estimation and Maximization](https://mlsanity.com/paper/2308.03950/) (2023)
- [CaDM: Codec-aware Diffusion Modeling for Neural-enhanced Video Streaming](https://mlsanity.com/paper/2211.08428/) (2022)
- [Towards Unbiased Multi-label Zero-Shot Learning with Pyramid and Semantic Attention](https://mlsanity.com/paper/2203.03483/) (2022)

## Built on by (most cited first, 19 of 19)

- [Neuron: Learning Context-Aware Evolving Representations for Zero-Shot Skeleton Action Recognition](https://mlsanity.com/paper/2411.11288/) (2024)
- [Zero-Shot Skeleton-based Action Recognition with Dual Visual-Text Alignment](https://mlsanity.com/paper/2409.14336/) (2024)
- [Bridging the Skeleton-Text Modality Gap: Diffusion-Powered Modality Alignment for Zero-shot Skeleton-based Action Recognition](https://mlsanity.com/paper/2411.10745/) (2024)
- [Frequency-Semantic Enhanced Variational Autoencoder for Zero-Shot Skeleton-based Action Recognition](https://mlsanity.com/paper/2506.22179/) (2025)
- [On the Element-Wise Representation and Reasoning in Zero-Shot Image Recognition: A Systematic Survey](https://mlsanity.com/paper/2408.04879/) (2024)
- [Towards Robust and Realistic Human Pose Estimation via WiFi Signals](https://mlsanity.com/paper/2501.09411/) (2025)
- [Dual Expert Distillation Network for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2404.16348/) (2024)
- [Epsilon: Exploring Comprehensive Visual-Semantic Projection for Multi-Label Zero-Shot Learning](https://mlsanity.com/paper/2408.12253/) (2024)
- [Learning by Neighbor-Aware Semantics, Deciding by Open-form Flows: Towards Robust Zero-Shot Skeleton Action Recognition](https://mlsanity.com/paper/2511.09388/) (2025)
- [Boosting Skeleton-based Zero-Shot Action Recognition with Training-Free Test-Time Adaptation](https://mlsanity.com/paper/2512.11458/) (2025)
- [Universal Skeleton Understanding via Differentiable Rendering and MLLMs](https://mlsanity.com/paper/2603.18003/) (2026)
- [SkeletonContext: Skeleton-side Context Prompt Learning for Zero-Shot Skeleton-based Action Recognition](https://mlsanity.com/paper/2603.29692/) (2026)
- [Frequency-Enhanced Diffusion Models: Curriculum-Guided Semantic Alignment for Zero-Shot Skeleton Action Recognition](https://mlsanity.com/paper/2604.09063/) (2026)
- [DynaPURLS: Dynamic Refinement of Part-Aware Representations for Skeleton-Based Zero-Shot Action Recognition](https://mlsanity.com/paper/2512.11941/) (2025)
- [SCALE: Semantic- and Confidence-Aware Conditional Variational Autoencoder for Zero-shot Skeleton-based Action Recognition](https://mlsanity.com/paper/2604.02222/) (2026)
- [Toward Universal Skeleton-Based Action Recognition across Heterogeneous Skeletons and Open Vocabularies](https://mlsanity.com/paper/2604.17013/) (2026)
- [PoseBridge: Bridging the Skeletonization Gap for Zero-Shot Skeleton-Based Action Recognition](https://mlsanity.com/paper/2605.11497/) (2026)
- [SkelHCC: A Hyperbolic CLIP-Driven Cache Adaptation Framework for Skeleton-based One-Shot Action Recognition](https://mlsanity.com/paper/2606.03610/) (2026)
- [GenPrior: Unleashing Text-to-Motion Generative Priors for Zero-Shot Skeleton-based Action Recognition](https://mlsanity.com/paper/2608.02236/) (2026)

## Abstract (arXiv)

Skeleton-based zero-shot action recognition aims to recognize unknown human actions based on the learned priors of the known skeleton-based actions and a semantic descriptor space shared by both known and unknown categories. However, previous works focus on establishing the bridges between the known skeleton representation space and semantic descriptions space at the coarse-grained level for recognizing unknown action categories, ignoring the fine-grained alignment of these two spaces, resulting in suboptimal performance in distinguishing high-similarity action categories. To address these challenges, we propose a novel method via Side information and dual-prompts learning for skeleton-based zero-shot action recognition (STAR) at the fine-grained level. Specifically, 1) we decompose the skeleton into several parts based on its topology structure and introduce the side information concerning multi-part descriptions of human body movements for alignment between the skeleton and the semantic space at the fine-grained level; 2) we design the visual-attribute and semantic-part prompts to improve the intra-class compactness within the skeleton space and inter-class separability within the semantic space, respectively, to distinguish the high-similarity actions. Extensive experiments show that our method achieves state-of-the-art performance in ZSL and GZSL settings on NTU RGB+D, NTU RGB+D 120, and PKU-MMD datasets.