mlsanityData to 14 Oct 2026

MaskSem: Semantic-Guided Masking for Learning 3D Hybrid High-Order Motion Representation

arXiv 2508.12948 ↗First posted 18 Aug 2025cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Human action recognition is a crucial task for intelligent robotics, particularly within the context of human-robot collaboration research. In self-supervised skeleton-based action recognition, the mask-based reconstruction paradigm learns the spatial structure and motion patterns of the skeleton by masking joints and reconstructing the target from unlabeled data. However, existing methods focus on a limited set of joints and low-order motion patterns, limiting the model's ability to understand complex motion patterns. To address this issue, we introduce MaskSem, a novel semantic-guided masking method for learning 3D hybrid high-order motion representations. This novel framework leverages Grad-CAM based on relative motion to guide the masking of joints, which can be represented as the most semantically rich temporal orgions. The semantic-guided masking process can encourage the model to explore more discriminative features. Furthermore, we propose using hybrid high-order motion as the reconstruction target, enabling the model to learn multi-order motion patterns. Specifically, low-order motion velocity and high-order motion acceleration are used together as the reconstruction target. This approach offers a more comprehensive description of the dynamic motion process, enhancing the model's understanding of motion patterns. Experiments on the NTU60, NTU120, and PKU-MMD datasets show that MaskSem, combined with a vanilla transformer, improves skeleton-based action recognition, making it more suitable for applications in human-robot interaction.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 44 references that are in the map. First posted on arXiv 18 Aug 2025.

Builds on

33 of 44 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  6. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  7. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,209
  8. 2016NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysiscited by 686
  9. 2019NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understandingcited by 533
  10. 2021Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognitioncited by 322
  11. 2022MotionBERT: A Unified Perspective on Learning Human Motion Representationscited by 193
  12. 2019An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognitioncited by 151
21 more
  1. 2016Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognitioncited by 143
  2. 20213D Human Action Representation Learning via Cross-View Consistency Pursuitcited by 106
  3. 2017PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understandingcited by 101
  4. 2020Skeleton-based Action Recognition via Spatial and Temporal Transformer Networkscited by 98
  5. 2019PREDICT & CLUSTER: Unsupervised Skeleton Based Action Recognitioncited by 91
  6. 2021Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-supervised Action Recognitioncited by 90
  7. 2021Skeleton-Contrastive 3D Action Representation Learningcited by 76
  8. 2021Skeleton Cloud Colorization for Unsupervised 3D Action Representation Learningcited by 58
  9. 2017Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gatescited by 54
  10. 2023Actionlet-Dependent Contrastive Learning for Unsupervised Skeleton-Based Action Recognitioncited by 42
  11. 2023SkeletonMAE: Graph-based Masked Autoencoder for Skeleton Sequence Pre-trainingcited by 41
  12. 2023Masked Motion Predictors are Strong 3D Action Representation Learnerscited by 41
  13. 2022CMD: Self-supervised 3D Action Representation Learning with Cross-modal Mutual Distillationcited by 38
  14. 2022Hierarchically Self-Supervised Transformer for Human Skeleton Representation Learningcited by 30
  15. 2022Hierarchical Contrast for Unsupervised Skeleton-based Action Representation Learningcited by 30
  16. 2022SkeletonMAE: Spatial-Temporal Masked Autoencoders for Self-supervised Skeleton Action Recognitioncited by 29
  17. 2023Self-supervised Action Representation Learning from Partial Spatio-Temporal Skeleton Sequencescited by 25
  18. 2023Interactive Spatiotemporal Token Attention Network for Skeleton-based General Interactive Action Recognitioncited by 23
  19. 2023Part Aware Contrastive Learning for Self-Supervised Action Recognitioncited by 21
  20. 2023Prompted Contrast with Masked Motion Modeling: Towards Versatile 3D Action Representation Learningcited by 19
  21. 2024Understanding Spatio-Temporal Relations in Human-Object Interaction using Pyramid Graph Convolutional Networkcited by 7

Built on by

Not cited yet.