# Spatio-Temporal Joint Density Driven Learning for Skeleton-Based Action Recognition

arXiv: https://arxiv.org/abs/2505.23012 · HTML: https://mlsanity.com/paper/2505.23012/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 55 references that are in the map.
- First posted on arXiv 29 May 2025.

## Builds on (most cited first, 30 of 40)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [RMPE: Regional Multi-person Pose Estimation](https://mlsanity.com/paper/1612.00137/) (2016)
- [S4L: Self-Supervised Semi-Supervised Learning](https://mlsanity.com/paper/1905.03670/) (2019)
- [Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2107.12213/) (2021)
- [Unsupervised Representation Learning by Sorting Sequences](https://mlsanity.com/paper/1708.01246/) (2017)
- [3D Human Action Representation Learning via Cross-View Consistency Pursuit](https://mlsanity.com/paper/2104.14466/) (2021)
- [PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding](https://mlsanity.com/paper/1703.07475/) (2017)
- [Self-supervised Spatio-temporal Representation Learning for Videos by Predicting Motion and Appearance Statistics](https://mlsanity.com/paper/1904.03597/) (2019)
- [MS2L: Multi-Task Self-Supervised Learning for Skeleton Based Action Recognition](https://mlsanity.com/paper/2010.05599/) (2020)
- [PREDICT & CLUSTER: Unsupervised Skeleton Based Action Recognition](https://mlsanity.com/paper/1911.12409/) (2019)
- [Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-supervised Action Recognition](https://mlsanity.com/paper/2112.03590/) (2021)
- [Skeleton-Contrastive 3D Action Representation Learning](https://mlsanity.com/paper/2108.03656/) (2021)
- [Augmented Skeleton Based Contrastive Action Learning with Momentum LSTM for Unsupervised Action Recognition](https://mlsanity.com/paper/2008.00188/) (2020)
- [Skeleton Cloud Colorization for Unsupervised 3D Action Representation Learning](https://mlsanity.com/paper/2108.01959/) (2021)
- [Actionlet-Dependent Contrastive Learning for Unsupervised Skeleton-Based Action Recognition](https://mlsanity.com/paper/2303.10904/) (2023)
- [SkeletonMAE: Graph-based Masked Autoencoder for Skeleton Sequence Pre-training](https://mlsanity.com/paper/2307.08476/) (2023)
- [Masked Motion Predictors are Strong 3D Action Representation Learners](https://mlsanity.com/paper/2308.07092/) (2023)
- [RGB-D-based Human Motion Recognition with Deep Learning: A Survey](https://mlsanity.com/paper/1711.08362/) (2017)

## Built on by (most cited first, 4 of 4)

- [Affinity Contrastive Learning for Skeleton-based Human Activity Understanding](https://mlsanity.com/paper/2601.16694/) (2026)
- [ASMa: Asymmetric Spatio-temporal Masking for Skeleton Action Representation Learning](https://mlsanity.com/paper/2602.06251/) (2026)
- [Subspace Kernel Learning on Tensor Sequences](https://mlsanity.com/paper/2603.19546/) (2026)
- [DiMoP: Diffusion-Driven Motion Representation Learning With Frame-Level Pseudo-Classification for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2609.35444/) (2026)

## Abstract (arXiv)

Traditional approaches in unsupervised or self supervised learning for skeleton-based action classification have concentrated predominantly on the dynamic aspects of skeletal sequences. Yet, the intricate interaction between the moving and static elements of the skeleton presents a rarely tapped discriminative potential for action classification. This paper introduces a novel measurement, referred to as spatial-temporal joint density (STJD), to quantify such interaction. Tracking the evolution of this density throughout an action can effectively identify a subset of discriminative moving and/or static joints termed "prime joints" to steer self-supervised learning. A new contrastive learning strategy named STJD-CL is proposed to align the representation of a skeleton sequence with that of its prime joints while simultaneously contrasting the representations of prime and nonprime joints. In addition, a method called STJD-MP is developed by integrating it with a reconstruction-based framework for more effective learning. Experimental evaluations on the NTU RGB+D 60, NTU RGB+D 120, and PKUMMD datasets in various downstream tasks demonstrate that the proposed STJD-CL and STJD-MP improved performance, particularly by 3.5 and 3.6 percentage points over the state-of-the-art contrastive methods on the NTU RGB+D 120 dataset using X-sub and X-set evaluations, respectively.