# An Information Compensation Framework for Zero-Shot Skeleton-based Action Recognition

arXiv: https://arxiv.org/abs/2406.00639 · HTML: https://mlsanity.com/paper/2406.00639/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 76 references that are in the map.
- First posted on arXiv 2 Jun 2024.

## Builds on (most cited first, 30 of 45)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://mlsanity.com/paper/2004.09813/) (2020)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2107.12213/) (2021)
- [Expanding Language-Image Pretrained Models for General Video Recognition](https://mlsanity.com/paper/2208.02816/) (2022)
- [ActionCLIP: A New Paradigm for Video Action Recognition](https://mlsanity.com/paper/2109.08472/) (2021)
- [Prompting Visual-Language Models for Efficient Video Understanding](https://mlsanity.com/paper/2112.04478/) (2021)
- [Large-Margin Softmax Loss for Convolutional Neural Networks](https://mlsanity.com/paper/1612.02295/) (2016)
- [What does a platypus look like? Generating customized prompts for zero-shot image classification](https://mlsanity.com/paper/2209.03320/) (2022)
- [Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders](https://mlsanity.com/paper/1812.01784/) (2018)
- [Co-occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregation](https://mlsanity.com/paper/1804.06055/) (2018)
- [3D Human Action Representation Learning via Cross-View Consistency Pursuit](https://mlsanity.com/paper/2104.14466/) (2021)
- [Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings](https://mlsanity.com/paper/1908.03477/) (2019)
- [Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-supervised Action Recognition](https://mlsanity.com/paper/2112.03590/) (2021)
- [View Adaptive Neural Networks for High Performance Skeleton-based Human Action Recognition](https://mlsanity.com/paper/1804.07453/) (2018)
- [PYSKL: Towards Good Practices for Skeleton Action Recognition](https://mlsanity.com/paper/2205.09443/) (2022)
- [Dynamic GCN: Context-enriched Topology Learning for Skeleton-based Action Recognition](https://mlsanity.com/paper/2007.14690/) (2020)
- [Rethinking Zero-shot Video Classification: End-to-end Training for Realistic Applications](https://mlsanity.com/paper/2003.01455/) (2020)
- [Elaborative Rehearsal for Zero-shot Action Recognition](https://mlsanity.com/paper/2108.02833/) (2021)
- [Revisiting Classifier: Transferring Vision-Language Models for Video Recognition](https://mlsanity.com/paper/2207.01297/) (2022)
- [Learning Robust Visual-Semantic Embeddings](https://mlsanity.com/paper/1703.05908/) (2017)

## Built on by (most cited first, 7 of 7)

- [Zero-Shot Skeleton-based Action Recognition with Dual Visual-Text Alignment](https://mlsanity.com/paper/2409.14336/) (2024)
- [Bridging the Skeleton-Text Modality Gap: Diffusion-Powered Modality Alignment for Zero-shot Skeleton-based Action Recognition](https://mlsanity.com/paper/2411.10745/) (2024)
- [Learning by Neighbor-Aware Semantics, Deciding by Open-form Flows: Towards Robust Zero-Shot Skeleton Action Recognition](https://mlsanity.com/paper/2511.09388/) (2025)
- [Frequency-Enhanced Diffusion Models: Curriculum-Guided Semantic Alignment for Zero-Shot Skeleton Action Recognition](https://mlsanity.com/paper/2604.09063/) (2026)
- [Prototype-Guided Curriculum Learning for Zero-Shot Learning](https://mlsanity.com/paper/2508.07771/) (2025)
- [Toward Universal Skeleton-Based Action Recognition across Heterogeneous Skeletons and Open Vocabularies](https://mlsanity.com/paper/2604.17013/) (2026)
- [OZ-TAL: Online Zero-Shot Temporal Action Localization](https://mlsanity.com/paper/2605.09976/) (2026)

## Abstract (arXiv)

Zero-shot human skeleton-based action recognition aims to construct a model that can recognize actions outside the categories seen during training. Previous research has focused on aligning sequences' visual and semantic spatial distributions. However, these methods extract semantic features simply. They ignore that proper prompt design for rich and fine-grained action cues can provide robust representation space clustering. In order to alleviate the problem of insufficient information available for skeleton sequences, we design an information compensation learning framework from an information-theoretic perspective to improve zero-shot action recognition accuracy with a multi-granularity semantic interaction mechanism. Inspired by ensemble learning, we propose a multi-level alignment (MLA) approach to compensate information for action classes. MLA aligns multi-granularity embeddings with visual embedding through a multi-head scoring mechanism to distinguish semantically similar action names and visually similar actions. Furthermore, we introduce a new loss function sampling method to obtain a tight and robust representation. Finally, these multi-granularity semantic embeddings are synthesized to form a proper decision surface for classification. Significant action recognition performance is achieved when evaluated on the challenging NTU RGB+D, NTU RGB+D 120, and PKU-MMD benchmarks and validate that multi-granularity semantic features facilitate the differentiation of action clusters with similar visual features.