mlsanityData to 7 Oct 2026

Heterogeneous Skeleton-Based Action Representation Learning

arXiv 2506.03481 ↗First posted 4 Jun 2025cs.CVcited by 12

Not on the map. Why?

Abstract (arXiv)

Skeleton-based human action recognition has received widespread attention in recent years due to its diverse range of application scenarios. Due to the different sources of human skeletons, skeleton data naturally exhibit heterogeneity. The previous works, however, overlook the heterogeneity of human skeletons and solely construct models tailored for homogeneous skeletons. This work addresses the challenge of heterogeneous skeleton-based action representation learning, specifically focusing on processing skeleton data that varies in joint dimensions and topological structures. The proposed framework comprises two primary components: heterogeneous skeleton processing and unified representation learning. The former first converts two-dimensional skeleton data into three-dimensional skeleton via an auxiliary network, and then constructs a prompted unified skeleton using skeleton-specific prompts. We also design an additional modality named semantic motion encoding to harness the semantic information within skeletons. The latter module learns a unified action representation using a shared backbone network that processes different heterogeneous skeletons. Extensive experiments on the NTU-60, NTU-120, and PKU-MMD II datasets demonstrate the effectiveness of our method in various tasks of action understanding. Our approach can be applied to action recognition in robots with different humanoid structures.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 44 of its 52 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

44 of 52 references.

References · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019Deep High-Resolution Representation Learning for Human Pose Estimationcited by 1,339
  3. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,199
  4. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  5. 2016NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysiscited by 683
  6. 2019NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understandingcited by 530
  7. 2020FineGym: A Hierarchical Video Dataset for Fine-grained Action Understandingcited by 232
  8. 2022MotionBERT: A Unified Perspective on Learning Human Motion Representationscited by 188
  9. 2019Video Representation Learning by Dense Predictive Codingcited by 175
  10. 20213D Human Action Representation Learning via Cross-View Consistency Pursuitcited by 105
  11. 2020Skeleton-based Action Recognition via Spatial and Temporal Transformer Networkscited by 98
  12. 2020MS2L: Multi-Task Self-Supervised Learning for Skeleton Based Action Recognitioncited by 92
32 more
  1. 2019PREDICT & CLUSTER: Unsupervised Skeleton Based Action Recognitioncited by 90
  2. 2021Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-supervised Action Recognitioncited by 89
  3. 2021Skeleton-Contrastive 3D Action Representation Learningcited by 76
  4. 2021Self-supervised Video Transformercited by 64
  5. 2021Skeleton Cloud Colorization for Unsupervised 3D Action Representation Learningcited by 58
  6. 2017Modeling Temporal Dynamics and Spatial Configurations of Actions Using Two-Stream Recurrent Neural Networkscited by 48
  7. 2021Learning the Predictability of the Futurecited by 48
  8. 2022Generative Action Description Prompts for Skeleton-based Action Recognitioncited by 43
  9. 2023HYperbolic Self-Paced Learning for Self-Supervised Skeleton-based Action Representationscited by 42
  10. 2023Actionlet-Dependent Contrastive Learning for Unsupervised Skeleton-Based Action Recognitioncited by 41
  11. 2023Masked Motion Predictors are Strong 3D Action Representation Learnerscited by 41
  12. 2022CMD: Self-supervised 3D Action Representation Learning with Cross-modal Mutual Distillationcited by 38
  13. 2022Hypergraph Transformer for Skeleton-based Action Recognitioncited by 36
  14. 2023MoLo: Motion-augmented Long-short Contrastive Learning for Few-shot Action Recognitioncited by 36
  15. 2020Adversarial Self-Supervised Learning for Semi-Supervised 3D Action Recognitioncited by 34
  16. 2022Contrastive Positive Mining for Unsupervised 3D Action Representation Learningcited by 34
  17. 2022Global-local Motion Transformer for Unsupervised Skeleton-based Action Learningcited by 33
  18. 2020Unsupervised 3D Human Pose Representation with Viewpoint and Pose Disentanglementcited by 32
  19. 2022Hierarchical Consistent Contrastive Learning for Skeleton-Based Action Recognition with Growing Augmentationscited by 31
  20. 2022Hierarchically Self-Supervised Transformer for Human Skeleton Representation Learningcited by 30
  21. 2022Hierarchical Contrast for Unsupervised Skeleton-based Action Representation Learningcited by 30
  22. 2022Frame-wise Action Representations for Long Videos via Sequence Contrastive Learningcited by 27
  23. 2023Self-supervised Action Representation Learning from Partial Spatio-Temporal Skeleton Sequencescited by 25
  24. 2023HaLP: Hallucinating Latent Positives for Skeleton-based Self-Supervised Learning of Actionscited by 25
  25. 2023Unified Multi-modal Unsupervised Representation Learning for Skeleton-based Action Understandingcited by 20
  26. 2021Contrast-reconstruction Representation Learning for Self-supervised Skeleton-based Action Recognitioncited by 19
  27. 2023Prompted Contrast with Masked Motion Modeling: Towards Versatile 3D Action Representation Learningcited by 19
  28. 2021Unsupervised Motion Representation Learning with Capsule Autoencoderscited by 16
  29. 2024USDRL: Unified Skeleton-Based Dense Representation Learning with Multi-Grained Feature Decorrelationcited by 15
  30. 2023A Large-scale Study of Spatiotemporal Representation Learning with a New Benchmark on Action Recognitioncited by 14
  31. 2023SCD-Net: Spatiotemporal Clues Disentanglement Network for Self-supervised Skeleton-based Action Recognitioncited by 11
  32. 2024Idempotent Unsupervised Representation Learning for Skeleton-Based Action Recognitioncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 8.

12 papers.

  1. 2025Foundation Model for Skeleton-Based Human Action Understandingcited by 11
  2. 2026Universal Skeleton Understanding via Differentiable Rendering and MLLMscited by 2
  3. 2025Heatmap Pooling Network for Action Recognition from RGB Videoscited by 1
  4. 2025Efficient Diffusion-Based 3D Human Pose Estimation with Hierarchical Temporal Pruningcited by 0
  5. 2025Skeleton-Snippet Contrastive Learning with Multiscale Feature Fusion for Action Localizationcited by 0
  6. 2026MotionRFT: Unified Reinforcement Fine-Tuning for Text-to-Motion Generationcited by 0
  7. 2026Toward Universal Skeleton-Based Action Recognition across Heterogeneous Skeletons and Open Vocabulariescited by 0
  8. 2026Beyond Binary Contrast: Modeling Continuous Skeleton Action Spaces with Transitional Anchorscited by 0
  9. 2026GenPrior: Unleashing Text-to-Motion Generative Priors for Zero-Shot Skeleton-based Action Recognitioncited by 0
  10. 2026Capturing Uncertainty in Human Motion for Representation Learning in Soccercited by 0
  11. 2026Zero-Shot Skeleton-Based Action Anticipationcited by 0
  12. 2026One for All: Generalist Foundation Model for Cross-Sensor Skeleton Representation Learningcited by 0