# Frequency-Semantic Enhanced Variational Autoencoder for Zero-Shot Skeleton-based Action Recognition

arXiv: https://arxiv.org/abs/2506.22179 · HTML: https://mlsanity.com/paper/2506.22179/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 51 references that are in the map.
- First posted on arXiv 27 Jun 2025.

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [In Defense of the Triplet Loss for Person Re-Identification](https://mlsanity.com/paper/1703.07737/) (2017)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [Deep metric learning using Triplet network](https://mlsanity.com/paper/1412.6622/) (2014)
- [Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2107.12213/) (2021)
- [3D Human Pose Estimation with Spatial and Temporal Transformers](https://mlsanity.com/paper/2103.10455/) (2021)
- [Learning Trajectory Dependencies for Human Motion Prediction](https://mlsanity.com/paper/1908.05436/) (2019)
- [Dynamic Multiscale Graph Neural Networks for 3D Skeleton-Based Human Motion Prediction](https://mlsanity.com/paper/2003.08802/) (2020)
- [A New Representation of Skeleton Sequences for 3D Action Recognition](https://mlsanity.com/paper/1703.03492/) (2017)
- [Deep Metric Learning with Hierarchical Triplet Loss](https://mlsanity.com/paper/1810.06951/) (2018)
- [PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding](https://mlsanity.com/paper/1703.07475/) (2017)
- [Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings](https://mlsanity.com/paper/1908.03477/) (2019)
- [PoseFormerV2: Exploring Frequency Domain for Efficient and Robust 3D Human Pose Estimation](https://mlsanity.com/paper/2303.17472/) (2023)
- [Multi-Person 3D Motion Prediction with Multi-Range Transformers](https://mlsanity.com/paper/2111.12073/) (2021)
- [Learning Robust Visual-Semantic Embeddings](https://mlsanity.com/paper/1703.05908/) (2017)
- [On Symmetric Losses for Learning from Corrupted Labels](https://mlsanity.com/paper/1901.09314/) (2019)
- [Learning Local Image Descriptors with Deep Siamese and Triplet Convolutional Networks by Minimising Global Loss Functions](https://mlsanity.com/paper/1512.09272/) (2015)
- [Skeleton-Parted Graph Scattering Networks for 3D Human Motion Prediction](https://mlsanity.com/paper/2208.00368/) (2022)
- [A Theoretically Sound Upper Bound on the Triplet Loss for Improving the Efficiency of Deep Distance Metric Learning](https://mlsanity.com/paper/1904.08720/) (2019)
- [Syntactically Guided Generative Embeddings for Zero-Shot Skeleton Action Recognition](https://mlsanity.com/paper/2101.11530/) (2021)
- [Zero-shot Skeleton-based Action Recognition via Mutual Information Estimation and Maximization](https://mlsanity.com/paper/2308.03950/) (2023)
- [Part-aware Unified Representation of Language and Skeleton for Zero-shot Action Recognition](https://mlsanity.com/paper/2406.13327/) (2024)
- [Fine-Grained Side Information Guided Dual-Prompts for Zero-Shot Skeleton Action Recognition](https://mlsanity.com/paper/2404.07487/) (2024)
- [SA-DVAE: Improving Zero-Shot Skeleton-Based Action Recognition by Disentangled Variational Autoencoders](https://mlsanity.com/paper/2407.13460/) (2024)
- [Frequency Guidance Matters: Skeletal Action Recognition by Frequency-Aware Mixed Transformer](https://mlsanity.com/paper/2407.12322/) (2024)
- [FeDXL: Provable Federated Learning for Deep X-Risk Optimization](https://mlsanity.com/paper/2210.14396/) (2022)

## Built on by (most cited first, 6 of 6)

- [Learning by Neighbor-Aware Semantics, Deciding by Open-form Flows: Towards Robust Zero-Shot Skeleton Action Recognition](https://mlsanity.com/paper/2511.09388/) (2025)
- [SkeletonContext: Skeleton-side Context Prompt Learning for Zero-Shot Skeleton-based Action Recognition](https://mlsanity.com/paper/2603.29692/) (2026)
- [Frequency-Enhanced Diffusion Models: Curriculum-Guided Semantic Alignment for Zero-Shot Skeleton Action Recognition](https://mlsanity.com/paper/2604.09063/) (2026)
- [Toward Universal Skeleton-Based Action Recognition across Heterogeneous Skeletons and Open Vocabularies](https://mlsanity.com/paper/2604.17013/) (2026)
- [PoseBridge: Bridging the Skeletonization Gap for Zero-Shot Skeleton-Based Action Recognition](https://mlsanity.com/paper/2605.11497/) (2026)
- [Visual Anchoring in Diffusion: Multimodal Zero-Shot Skeleton Action Recognition](https://mlsanity.com/paper/2608.04623/) (2026)

## Abstract (arXiv)

Zero-shot skeleton-based action recognition aims to develop models capable of identifying actions beyond the categories encountered during training. Previous approaches have primarily focused on aligning visual and semantic representations but often overlooked the importance of fine-grained action patterns in the semantic space (e.g., the hand movements in drinking water and brushing teeth). To address these limitations, we propose a Frequency-Semantic Enhanced Variational Autoencoder (FS-VAE) to explore the skeleton semantic representation learning with frequency decomposition. FS-VAE consists of three key components: 1) a frequency-based enhancement module with high- and low-frequency adjustments to enrich the skeletal semantics learning and improve the robustness of zero-shot action recognition; 2) a semantic-based action description with multilevel alignment to capture both local details and global correspondence, effectively bridging the semantic gap and compensating for the inherent loss of information in skeleton sequences; 3) a calibrated cross-alignment loss that enables valid skeleton-text pairs to counterbalance ambiguous ones, mitigating discrepancies and ambiguities in skeleton and text features, thereby ensuring robust alignment. Evaluations on the benchmarks demonstrate the effectiveness of our approach, validating that frequency-enhanced semantic features enable robust differentiation of visually and semantically similar action clusters, improving zero-shot action recognition.