# Part Aware Contrastive Learning for Self-Supervised Action Recognition

arXiv: https://arxiv.org/abs/2305.00666 · HTML: https://mlsanity.com/paper/2305.00666/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 51 references that are in the map.
- First posted on arXiv 1 May 2023.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [Contrastive Clustering](https://mlsanity.com/paper/2009.09687/) (2020)
- [Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognition](https://mlsanity.com/paper/1904.12659/) (2019)
- [Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition](https://mlsanity.com/paper/1904.01189/) (2019)
- [A New Representation of Skeleton Sequences for 3D Action Recognition](https://mlsanity.com/paper/1703.03492/) (2017)
- [3D Human Action Representation Learning via Cross-View Consistency Pursuit](https://mlsanity.com/paper/2104.14466/) (2021)
- [Self-Guided Contrastive Learning for BERT Sentence Representations](https://mlsanity.com/paper/2106.07345/) (2021)
- [BYOL works even without batch statistics](https://mlsanity.com/paper/2010.10241/) (2020)
- [PREDICT & CLUSTER: Unsupervised Skeleton Based Action Recognition](https://mlsanity.com/paper/1911.12409/) (2019)
- [Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-supervised Action Recognition](https://mlsanity.com/paper/2112.03590/) (2021)
- [Spatial Temporal Transformer Network for Skeleton-based Action Recognition](https://mlsanity.com/paper/2012.06399/) (2020)
- [Skeleton-based Action Recognition with Convolutional Neural Networks](https://mlsanity.com/paper/1704.07595/) (2017)
- [Iterative Reorganization with Weak Spatial Constraints: Solving Arbitrary Jigsaw Puzzles for Unsupervised Representation Learning](https://mlsanity.com/paper/1812.00329/) (2018)
- [Learning Where to Learn in Cross-View Self-Supervised Learning](https://mlsanity.com/paper/2203.14898/) (2022)
- [Hierarchical Consistent Contrastive Learning for Skeleton-Based Action Recognition with Growing Augmentations](https://mlsanity.com/paper/2211.13466/) (2022)

## Built on by (most cited first, 21 of 21)

- [SCD-Net: Spatiotemporal Clues Disentanglement Network for Self-supervised Skeleton-based Action Recognition](https://mlsanity.com/paper/2309.05834/) (2023)
- [Foundation Model for Skeleton-Based Human Action Understanding](https://mlsanity.com/paper/2508.12586/) (2025)
- [Human Pose-based Estimation, Tracking and Action Recognition with Deep Learning: A Survey](https://mlsanity.com/paper/2310.13039/) (2023)
- [RoMo: A Robust Solver for Full-body Unlabeled Optical Motion Capture](https://mlsanity.com/paper/2410.02788/) (2024)
- [3D Skeleton-Based Action Recognition: A Review](https://mlsanity.com/paper/2506.00915/) (2025)
- [A Locality-based Neural Solver for Optical Motion Capture](https://mlsanity.com/paper/2309.00428/) (2023)
- [Self-Supervised Skeleton-Based Action Representation Learning: A Benchmark and Beyond](https://mlsanity.com/paper/2406.02978/) (2024)
- [Shap-Mix: Shapley Value Guided Mixing for Long-Tailed Skeleton Based Action Recognition](https://mlsanity.com/paper/2407.12312/) (2024)
- [LAMP: Leveraging Language Prompts for Multi-person Pose Estimation](https://mlsanity.com/paper/2307.11934/) (2023)
- [STARS: Self-supervised Tuning for 3D Action Recognition in Skeleton Sequences](https://mlsanity.com/paper/2407.10935/) (2024)
- [KAN-HyperpointNet for Point Cloud Sequence-Based 3D Human Action Recognition](https://mlsanity.com/paper/2409.09444/) (2024)
- [Towards Efficient General Feature Prediction in Masked Skeleton Modeling](https://mlsanity.com/paper/2509.03609/) (2025)
- [Skeleton2vec: A Self-supervised Learning Framework with Contextualized Target Representations for Skeleton Sequence](https://mlsanity.com/paper/2401.00921/) (2024)
- [Content-Distortion High-Order Interaction for Blind Image Quality Assessment](https://mlsanity.com/paper/2504.05076/) (2025)
- [Multimodal Skeleton-Based Action Representation Learning via Decomposition and Composition](https://mlsanity.com/paper/2512.21064/) (2025)
- [Spatiotemporal-Linear: Towards Universal Multivariate Time Series Forecasting](https://mlsanity.com/paper/2312.14869/) (2023)
- [Spatial Hierarchy and Temporal Attention Guided Cross Masking for Self-supervised Skeleton-based Action Recognition](https://mlsanity.com/paper/2409.17951/) (2024)
- [MaskSem: Semantic-Guided Masking for Learning 3D Hybrid High-Order Motion Representation](https://mlsanity.com/paper/2508.12948/) (2025)
- [DoGCLR: Dominance-Game Contrastive Learning Network for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2511.14179/) (2025)
- [M3GCLR: Multi-View Mini-Max Infinite Skeleton-Data Game Contrastive Learning For Skeleton-Based Action Recognition](https://mlsanity.com/paper/2603.09367/) (2026)
- [Beyond Binary Contrast: Modeling Continuous Skeleton Action Spaces with Transitional Anchors](https://mlsanity.com/paper/2604.17914/) (2026)

## Abstract (arXiv)

In recent years, remarkable results have been achieved in self-supervised action recognition using skeleton sequences with contrastive learning. It has been observed that the semantic distinction of human action features is often represented by local body parts, such as legs or hands, which are advantageous for skeleton-based action recognition. This paper proposes an attention-based contrastive learning framework for skeleton representation learning, called SkeAttnCLR, which integrates local similarity and global features for skeleton-based action representations. To achieve this, a multi-head attention mask module is employed to learn the soft attention mask features from the skeletons, suppressing non-salient local features while accentuating local salient features, thereby bringing similar local features closer in the feature space. Additionally, ample contrastive pairs are generated by expanding contrastive pairs based on salient and non-salient features with global features, which guide the network to learn the semantic representations of the entire skeleton. Therefore, with the attention mask mechanism, SkeAttnCLR learns local features under different data augmentation views. The experiment results demonstrate that the inclusion of local feature similarity significantly enhances skeleton-based action representation. Our proposed SkeAttnCLR outperforms state-of-the-art methods on NTURGB+D, NTU120-RGB+D, and PKU-MMD datasets.