# Contrastive Learning from Spatio-Temporal Mixed Skeleton Sequences for Self-Supervised Skeleton-Based Action Recognition

arXiv: https://arxiv.org/abs/2207.03065 · HTML: https://mlsanity.com/paper/2207.03065/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 52 references that are in the map.
- First posted on arXiv 7 Jul 2022.

## Builds on (most cited first, 30 of 33)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [Hard Negative Mixing for Contrastive Learning](https://mlsanity.com/paper/2010.01028/) (2020)
- [Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2003.14111/) (2020)
- [Contrastive Clustering](https://mlsanity.com/paper/2009.09687/) (2020)
- [Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognition](https://mlsanity.com/paper/1904.12659/) (2019)
- [Human Action Recognition from Various Data Modalities: A Review](https://mlsanity.com/paper/2012.11866/) (2020)
- [MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimation](https://mlsanity.com/paper/2111.12707/) (2021)
- [3D Human Action Representation Learning via Cross-View Consistency Pursuit](https://mlsanity.com/paper/2104.14466/) (2021)
- [Exploiting Temporal Contexts with Strided Transformer for 3D Human Pose Estimation](https://mlsanity.com/paper/2103.14304/) (2021)
- [MS2L: Multi-Task Self-Supervised Learning for Skeleton Based Action Recognition](https://mlsanity.com/paper/2010.05599/) (2020)
- [PREDICT & CLUSTER: Unsupervised Skeleton Based Action Recognition](https://mlsanity.com/paper/1911.12409/) (2019)
- [Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-supervised Action Recognition](https://mlsanity.com/paper/2112.03590/) (2021)
- [Skeleton-Contrastive 3D Action Representation Learning](https://mlsanity.com/paper/2108.03656/) (2021)
- [Un-Mix: Rethinking Image Mixtures for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/2003.05438/) (2020)
- [MixCo: Mix-up Contrastive Learning for Visual Representation](https://mlsanity.com/paper/2010.06300/) (2020)
- [Augmented Skeleton Based Contrastive Action Learning with Momentum LSTM for Unsupervised Action Recognition](https://mlsanity.com/paper/2008.00188/) (2020)
- [Skeleton Cloud Colorization for Unsupervised 3D Action Representation Learning](https://mlsanity.com/paper/2108.01959/) (2021)

## Built on by (most cited first, 17 of 17)

- [Actionlet-Dependent Contrastive Learning for Unsupervised Skeleton-Based Action Recognition](https://mlsanity.com/paper/2303.10904/) (2023)
- [Part Aware Contrastive Learning for Self-Supervised Action Recognition](https://mlsanity.com/paper/2305.00666/) (2023)
- [Prompted Contrast with Masked Motion Modeling: Towards Versatile 3D Action Representation Learning](https://mlsanity.com/paper/2308.03975/) (2023)
- [MacDiff: Unified Skeleton Modeling with Masked Conditional Diffusion](https://mlsanity.com/paper/2409.10473/) (2024)
- [Identity-free Artificial Emotional Intelligence via Micro-Gesture Understanding](https://mlsanity.com/paper/2405.13206/) (2024)
- [Idempotent Unsupervised Representation Learning for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2410.20349/) (2024)
- [DMMG: Dual Min-Max Games for Self-Supervised Skeleton-Based Action Recognition](https://mlsanity.com/paper/2302.12007/) (2023)
- [Self-Supervised Skeleton-Based Action Representation Learning: A Benchmark and Beyond](https://mlsanity.com/paper/2406.02978/) (2024)
- [Shap-Mix: Shapley Value Guided Mixing for Long-Tailed Skeleton Based Action Recognition](https://mlsanity.com/paper/2407.12312/) (2024)
- [Spatio-Temporal Joint Density Driven Learning for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2505.23012/) (2025)
- [Balanced Representation Learning for Long-tailed Skeleton-based Action Recognition](https://mlsanity.com/paper/2308.14024/) (2023)
- [DuoCLR: Dual-Surrogate Contrastive Learning for Skeleton-based Human Action Segmentation](https://mlsanity.com/paper/2509.05543/) (2025)
- [MS-CLR: Multi-Skeleton Contrastive Learning for Human Action Recognition](https://mlsanity.com/paper/2508.14889/) (2025)
- [DoGCLR: Dominance-Game Contrastive Learning Network for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2511.14179/) (2025)
- [M3GCLR: Multi-View Mini-Max Infinite Skeleton-Data Game Contrastive Learning For Skeleton-Based Action Recognition](https://mlsanity.com/paper/2603.09367/) (2026)
- [Beyond Binary Contrast: Modeling Continuous Skeleton Action Spaces with Transitional Anchors](https://mlsanity.com/paper/2604.17914/) (2026)
- [DiMoP: Diffusion-Driven Motion Representation Learning With Frame-Level Pseudo-Classification for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2609.35444/) (2026)

## Abstract (arXiv)

Self-supervised skeleton-based action recognition with contrastive learning has attracted much attention. Recent literature shows that data augmentation and large sets of contrastive pairs are crucial in learning such representations. In this paper, we found that directly extending contrastive pairs based on normal augmentations brings limited returns in terms of performance, because the contribution of contrastive pairs from the normal data augmentation to the loss get smaller as training progresses. Therefore, we delve into hard contrastive pairs for contrastive learning. Motivated by the success of mixing augmentation strategy which improves the performance of many tasks by synthesizing novel samples, we propose SkeleMixCLR: a contrastive learning framework with a spatio-temporal skeleton mixing augmentation (SkeleMix) to complement current contrastive learning approaches by providing hard contrastive samples. First, SkeleMix utilizes the topological information of skeleton data to mix two skeleton sequences by randomly combing the cropped skeleton fragments (the trimmed view) with the remaining skeleton sequences (the truncated view). Second, a spatio-temporal mask pooling is applied to separate these two views at the feature level. Third, we extend contrastive pairs with these two views. SkeleMixCLR leverages the trimmed and truncated views to provide abundant hard contrastive pairs since they involve some context information from each other due to the graph convolution operations, which allows the model to learn better motion representations for action recognition. Extensive experiments on NTU-RGB+D, NTU120-RGB+D, and PKU-MMD datasets show that SkeleMixCLR achieves state-of-the-art performance. Codes are available at https://github.com/czhaneva/SkeleMixCLR.