mlsanityData to 14 Oct 2026

Spatial Hierarchy and Temporal Attention Guided Cross Masking for Self-supervised Skeleton-based Action Recognition

arXiv 2409.17951 ↗First posted 26 Sep 2024cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

In self-supervised skeleton-based action recognition, the mask reconstruction paradigm is gaining interest in enhancing model refinement and robustness through effective masking. However, previous works primarily relied on a single masking criterion, resulting in the model overfitting specific features and overlooking other effective information. In this paper, we introduce a hierarchy and attention guided cross-masking framework (HA-CM) that applies masking to skeleton sequences from both spatial and temporal perspectives. Specifically, in spatial graphs, we utilize hyperbolic space to maintain joint distinctions and effectively preserve the hierarchical structure of high-dimensional skeletons, employing joint hierarchy as the masking criterion. In temporal flows, we substitute traditional distance metrics with the global attention of joints for masking, addressing the convergence of distances in high-dimensional space and the lack of a global perspective. Additionally, we incorporate cross-contrast loss based on the cross-masking framework into the loss function to enhance the model's learning of instance-level features. HA-CM shows efficiency and universality on three public large-scale datasets, NTU-60, NTU-120, and PKU-MMD. The source code of our HA-CM is available at https://github.com/YinxPeng/HA-CM-main.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 30 of its 41 references that are in the map. First posted on arXiv 26 Sep 2024.

Builds on

30 of 41 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  3. 2016NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysiscited by 686
  4. 2019NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understandingcited by 533
  5. 2018Hyperbolic Neural Networkscited by 402
  6. 2019Hyperbolic Graph Neural Networkscited by 244
  7. 2022MotionBERT: A Unified Perspective on Learning Human Motion Representationscited by 193
  8. 20213D Human Action Representation Learning via Cross-View Consistency Pursuitcited by 106
  9. 2017PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understandingcited by 101
  10. 2019PREDICT & CLUSTER: Unsupervised Skeleton Based Action Recognitioncited by 91
  11. 2021Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-supervised Action Recognitioncited by 90
  12. 2021Skeleton-Contrastive 3D Action Representation Learningcited by 76
18 more
  1. 2020Augmented Skeleton Based Contrastive Action Learning with Momentum LSTM for Unsupervised Action Recognitioncited by 60
  2. 2019Spatiotemporal Co-attention Recurrent Neural Networks for Human-Skeleton Motion Predictioncited by 47
  3. 2023HYperbolic Self-Paced Learning for Self-Supervised Skeleton-based Action Representationscited by 44
  4. 2023Actionlet-Dependent Contrastive Learning for Unsupervised Skeleton-Based Action Recognitioncited by 42
  5. 2023SkeletonMAE: Graph-based Masked Autoencoder for Skeleton Sequence Pre-trainingcited by 41
  6. 2023Masked Motion Predictors are Strong 3D Action Representation Learnerscited by 41
  7. 2022CMD: Self-supervised 3D Action Representation Learning with Cross-modal Mutual Distillationcited by 38
  8. 2022Hierarchical Consistent Contrastive Learning for Skeleton-Based Action Recognition with Growing Augmentationscited by 31
  9. 2022Hierarchical Contrast for Unsupervised Skeleton-based Action Representation Learningcited by 30
  10. 2022SkeletonMAE: Spatial-Temporal Masked Autoencoders for Self-supervised Skeleton Action Recognitioncited by 29
  11. 2023HaLP: Hallucinating Latent Positives for Skeleton-based Self-Supervised Learning of Actionscited by 25
  12. 2023Part Aware Contrastive Learning for Self-Supervised Action Recognitioncited by 21
  13. 2023Prompted Contrast with Masked Motion Modeling: Towards Versatile 3D Action Representation Learningcited by 19
  14. 2023Self-Supervised 3D Action Representation Learning with Skeleton Cloud Colorizationcited by 16
  15. 2017Skeleton-aided Articulated Motion Generationcited by 14
  16. 2023SCD-Net: Spatiotemporal Clues Disentanglement Network for Self-supervised Skeleton-based Action Recognitioncited by 11
  17. 2023Dynamic Hyperbolic Attention Network for Fine Hand-object Reconstructioncited by 10
  18. 2024Skeleton2vec: A Self-supervised Learning Framework with Contextualized Target Representations for Skeleton Sequencecited by 2

Built on by

Not cited yet.