# Multi-Semantic Fusion Model for Generalized Zero-Shot Skeleton-Based Action Recognition

arXiv: https://arxiv.org/abs/2309.09592 · HTML: https://mlsanity.com/paper/2309.09592/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 18 Sep 2023.

## Builds on (most cited first, 18 of 18)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2003.14111/) (2020)
- [Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders](https://mlsanity.com/paper/1812.01784/) (2018)
- [A Review of Generalized Zero-Shot Learning Methods](https://mlsanity.com/paper/2011.08641/) (2020)
- [Semantic Autoencoder for Zero-Shot Learning](https://mlsanity.com/paper/1704.08345/) (2017)
- [Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings](https://mlsanity.com/paper/1908.03477/) (2019)
- [Constructing Stronger and Faster Baselines for Skeleton-based Action Recognition](https://mlsanity.com/paper/2106.15125/) (2021)
- [View Adaptive Neural Networks for High Performance Skeleton-based Human Action Recognition](https://mlsanity.com/paper/1804.07453/) (2018)
- [Goal-Oriented Gaze Estimation for Zero-Shot Learning](https://mlsanity.com/paper/2103.03433/) (2021)
- [Learning Robust Visual-Semantic Embeddings](https://mlsanity.com/paper/1703.05908/) (2017)
- [Syntactically Guided Generative Embeddings for Zero-Shot Skeleton Action Recognition](https://mlsanity.com/paper/2101.11530/) (2021)
- [Towards Effective Deep Embedding for Zero-Shot Learning](https://mlsanity.com/paper/1808.10075/) (2018)
- [Skeleton based Zero Shot Action Recognition in Joint Pose-Language Semantic Space](https://mlsanity.com/paper/1911.11344/) (2019)

## Built on by (most cited first, 9 of 9)

- [Fine-Grained Side Information Guided Dual-Prompts for Zero-Shot Skeleton Action Recognition](https://mlsanity.com/paper/2404.07487/) (2024)
- [Neuron: Learning Context-Aware Evolving Representations for Zero-Shot Skeleton Action Recognition](https://mlsanity.com/paper/2411.11288/) (2024)
- [Bridging the Skeleton-Text Modality Gap: Diffusion-Powered Modality Alignment for Zero-shot Skeleton-based Action Recognition](https://mlsanity.com/paper/2411.10745/) (2024)
- [An Information Compensation Framework for Zero-Shot Skeleton-based Action Recognition](https://mlsanity.com/paper/2406.00639/) (2024)
- [Frequency-Semantic Enhanced Variational Autoencoder for Zero-Shot Skeleton-based Action Recognition](https://mlsanity.com/paper/2506.22179/) (2025)
- [Learning by Neighbor-Aware Semantics, Deciding by Open-form Flows: Towards Robust Zero-Shot Skeleton Action Recognition](https://mlsanity.com/paper/2511.09388/) (2025)
- [SkeletonContext: Skeleton-side Context Prompt Learning for Zero-Shot Skeleton-based Action Recognition](https://mlsanity.com/paper/2603.29692/) (2026)
- [Frequency-Enhanced Diffusion Models: Curriculum-Guided Semantic Alignment for Zero-Shot Skeleton Action Recognition](https://mlsanity.com/paper/2604.09063/) (2026)
- [SCALE: Semantic- and Confidence-Aware Conditional Variational Autoencoder for Zero-shot Skeleton-based Action Recognition](https://mlsanity.com/paper/2604.02222/) (2026)

## Abstract (arXiv)

Generalized zero-shot skeleton-based action recognition (GZSSAR) is a new challenging problem in computer vision community, which requires models to recognize actions without any training samples. Previous studies only utilize the action labels of verb phrases as the semantic prototypes for learning the mapping from skeleton-based actions to a shared semantic space. However, the limited semantic information of action labels restricts the generalization ability of skeleton features for recognizing unseen actions. In order to solve this dilemma, we propose a multi-semantic fusion (MSF) model for improving the performance of GZSSAR, where two kinds of class-level textual descriptions (i.e., action descriptions and motion descriptions), are collected as auxiliary semantic information to enhance the learning efficacy of generalizable skeleton features. Specially, a pre-trained language encoder takes the action descriptions, motion descriptions and original class labels as inputs to obtain rich semantic features for each action class, while a skeleton encoder is implemented to extract skeleton features. Then, a variational autoencoder (VAE) based generative module is performed to learn a cross-modal alignment between skeleton and semantic features. Finally, a classification module is built to recognize the action categories of input samples, where a seen-unseen classification gate is adopted to predict whether the sample comes from seen action classes or not in GZSSAR. The superior performance in comparisons with previous models validates the effectiveness of the proposed MSF model on GZSSAR.