# SCD-Net: Spatiotemporal Clues Disentanglement Network for Self-supervised Skeleton-based Action Recognition

arXiv: https://arxiv.org/abs/2309.05834 · HTML: https://mlsanity.com/paper/2309.05834/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 55 references that are in the map.
- First posted on arXiv 11 Sep 2023.

## Builds on (most cited first, 30 of 38)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2003.14111/) (2020)
- [Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2107.12213/) (2021)
- [Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognition](https://mlsanity.com/paper/1904.12659/) (2019)
- [TDN: Temporal Difference Networks for Efficient Action Recognition](https://mlsanity.com/paper/2012.10071/) (2020)
- [A New Representation of Skeleton Sequences for 3D Action Recognition](https://mlsanity.com/paper/1703.03492/) (2017)
- [3D Human Action Representation Learning via Cross-View Consistency Pursuit](https://mlsanity.com/paper/2104.14466/) (2021)
- [PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding](https://mlsanity.com/paper/1703.07475/) (2017)
- [MS2L: Multi-Task Self-Supervised Learning for Skeleton Based Action Recognition](https://mlsanity.com/paper/2010.05599/) (2020)
- [PREDICT & CLUSTER: Unsupervised Skeleton Based Action Recognition](https://mlsanity.com/paper/1911.12409/) (2019)
- [Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-supervised Action Recognition](https://mlsanity.com/paper/2112.03590/) (2021)
- [PYSKL: Towards Good Practices for Skeleton Action Recognition](https://mlsanity.com/paper/2205.09443/) (2022)
- [Skeleton-Contrastive 3D Action Representation Learning](https://mlsanity.com/paper/2108.03656/) (2021)
- [S3VAE: Self-Supervised Sequential VAE for Representation Disentanglement and Data Generation](https://mlsanity.com/paper/2005.11437/) (2020)
- [Skeleton Cloud Colorization for Unsupervised 3D Action Representation Learning](https://mlsanity.com/paper/2108.01959/) (2021)
- [HYperbolic Self-Paced Learning for Self-Supervised Skeleton-based Action Representations](https://mlsanity.com/paper/2303.06242/) (2023)
- [Actionlet-Dependent Contrastive Learning for Unsupervised Skeleton-Based Action Recognition](https://mlsanity.com/paper/2303.10904/) (2023)
- [Adversarial Self-Supervised Learning for Semi-Supervised 3D Action Recognition](https://mlsanity.com/paper/2007.05934/) (2020)
- [Contrastive Positive Mining for Unsupervised 3D Action Representation Learning](https://mlsanity.com/paper/2208.03497/) (2022)
- [Global-local Motion Transformer for Unsupervised Skeleton-based Action Learning](https://mlsanity.com/paper/2207.06101/) (2022)

## Built on by (most cited first, 11 of 11)

- [Heterogeneous Skeleton-Based Action Representation Learning](https://mlsanity.com/paper/2506.03481/) (2025)
- [Foundation Model for Skeleton-Based Human Action Understanding](https://mlsanity.com/paper/2508.12586/) (2025)
- [3D Skeleton-Based Action Recognition: A Review](https://mlsanity.com/paper/2506.00915/) (2025)
- [DuoCLR: Dual-Surrogate Contrastive Learning for Skeleton-based Human Action Segmentation](https://mlsanity.com/paper/2509.05543/) (2025)
- [Multimodal Skeleton-Based Action Representation Learning via Decomposition and Composition](https://mlsanity.com/paper/2512.21064/) (2025)
- [Spatial Hierarchy and Temporal Attention Guided Cross Masking for Self-supervised Skeleton-based Action Recognition](https://mlsanity.com/paper/2409.17951/) (2024)
- [DoGCLR: Dominance-Game Contrastive Learning Network for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2511.14179/) (2025)
- [ASMa: Asymmetric Spatio-temporal Masking for Skeleton Action Representation Learning](https://mlsanity.com/paper/2602.06251/) (2026)
- [M3GCLR: Multi-View Mini-Max Infinite Skeleton-Data Game Contrastive Learning For Skeleton-Based Action Recognition](https://mlsanity.com/paper/2603.09367/) (2026)
- [Toward Universal Skeleton-Based Action Recognition across Heterogeneous Skeletons and Open Vocabularies](https://mlsanity.com/paper/2604.17013/) (2026)
- [Action Motifs: Self-Supervised Hierarchical Representation of Human Body Movements](https://mlsanity.com/paper/2604.28173/) (2026)

## Abstract (arXiv)

Contrastive learning has achieved great success in skeleton-based action recognition. However, most existing approaches encode the skeleton sequences as entangled spatiotemporal representations and confine the contrasts to the same level of representation. Instead, this paper introduces a novel contrastive learning framework, namely Spatiotemporal Clues Disentanglement Network (SCD-Net). Specifically, we integrate the decoupling module with a feature extractor to derive explicit clues from spatial and temporal domains respectively. As for the training of SCD-Net, with a constructed global anchor, we encourage the interaction between the anchor and extracted clues. Further, we propose a new masking strategy with structural constraints to strengthen the contextual associations, leveraging the latest development from masked image modelling into the proposed SCD-Net. We conduct extensive evaluations on the NTU-RGB+D (60&120) and PKU-MMD (I&II) datasets, covering various downstream tasks such as action recognition, action retrieval, transfer learning, and semi-supervised learning. The experimental results demonstrate the effectiveness of our method, which outperforms the existing state-of-the-art (SOTA) approaches significantly.