# Augmented Skeleton Based Contrastive Action Learning with Momentum LSTM for Unsupervised Action Recognition

arXiv: https://arxiv.org/abs/2008.00188 · HTML: https://mlsanity.com/paper/2008.00188/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Aug 2020.

## Built on by (most cited first, 30 of 60)

- [3D Human Action Representation Learning via Cross-View Consistency Pursuit](https://mlsanity.com/paper/2104.14466/) (2021)
- [Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-supervised Action Recognition](https://mlsanity.com/paper/2112.03590/) (2021)
- [HYperbolic Self-Paced Learning for Self-Supervised Skeleton-based Action Representations](https://mlsanity.com/paper/2303.06242/) (2023)
- [Actionlet-Dependent Contrastive Learning for Unsupervised Skeleton-Based Action Recognition](https://mlsanity.com/paper/2303.10904/) (2023)
- [SkeletonMAE: Graph-based Masked Autoencoder for Skeleton Sequence Pre-training](https://mlsanity.com/paper/2307.08476/) (2023)
- [Masked Motion Predictors are Strong 3D Action Representation Learners](https://mlsanity.com/paper/2308.07092/) (2023)
- [CMD: Self-supervised 3D Action Representation Learning with Cross-modal Mutual Distillation](https://mlsanity.com/paper/2208.12448/) (2022)
- [Contrastive Positive Mining for Unsupervised 3D Action Representation Learning](https://mlsanity.com/paper/2208.03497/) (2022)
- [Global-local Motion Transformer for Unsupervised Skeleton-based Action Learning](https://mlsanity.com/paper/2207.06101/) (2022)
- [Hierarchical Consistent Contrastive Learning for Skeleton-Based Action Recognition with Growing Augmentations](https://mlsanity.com/paper/2211.13466/) (2022)
- [Hierarchical Contrast for Unsupervised Skeleton-based Action Representation Learning](https://mlsanity.com/paper/2212.02082/) (2022)
- [More than Encoder: Introducing Transformer Decoder to Upsample](https://mlsanity.com/paper/2106.10637/) (2021)
- [Self-supervised Action Representation Learning from Partial Spatio-Temporal Skeleton Sequences](https://mlsanity.com/paper/2302.09018/) (2023)
- [HaLP: Hallucinating Latent Positives for Skeleton-based Self-Supervised Learning of Actions](https://mlsanity.com/paper/2304.00387/) (2023)
- [ViewCLR: Learning Self-supervised Video Representation for Unseen Viewpoints](https://mlsanity.com/paper/2112.03905/) (2021)
- [Unified Multi-modal Unsupervised Representation Learning for Skeleton-based Action Understanding](https://mlsanity.com/paper/2311.03106/) (2023)
- [Contrast-reconstruction Representation Learning for Self-supervised Skeleton-based Action Recognition](https://mlsanity.com/paper/2111.11051/) (2021)
- [IIP-Transformer: Intra-Inter-Part Transformer for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2110.13385/) (2021)
- [Contrastive Learning from Spatio-Temporal Mixed Skeleton Sequences for Self-Supervised Skeleton-Based Action Recognition](https://mlsanity.com/paper/2207.03065/) (2022)
- [Spatiotemporal Decouple-and-Squeeze Contrastive Learning for Semi-Supervised Skeleton-based Action Recognition](https://mlsanity.com/paper/2302.02316/) (2023)
- [Unsupervised Motion Representation Learning with Capsule Autoencoders](https://mlsanity.com/paper/2110.00529/) (2021)
- [Self-Supervised 3D Action Representation Learning with Skeleton Cloud Colorization](https://mlsanity.com/paper/2304.08799/) (2023)
- [MacDiff: Unified Skeleton Modeling with Masked Conditional Diffusion](https://mlsanity.com/paper/2409.10473/) (2024)
- [TranSG: Transformer-Based Skeleton Graph Prototype Contrastive Learning with Structure-Trajectory Prompted Reconstruction for Person Re-Identification](https://mlsanity.com/paper/2303.06819/) (2023)
- [Pyramid Self-attention Polymerization Learning for Semi-supervised Skeleton-based Action Recognition](https://mlsanity.com/paper/2302.02327/) (2023)
- [Unsupervised Human Action Recognition with Skeletal Graph Laplacian and Self-Supervised Viewpoints Invariance](https://mlsanity.com/paper/2204.10312/) (2022)
- [Skeleton-based Action Recognition through Contrasting Two-Stream Spatial-Temporal Networks](https://mlsanity.com/paper/2301.11495/) (2023)
- [Bootstrapped Representation Learning for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2202.02232/) (2022)
- [Identity-free Artificial Emotional Intelligence via Micro-Gesture Understanding](https://mlsanity.com/paper/2405.13206/) (2024)
- [Multimodal Emotion Recognition with Modality-Pairwise Unsupervised Contrastive Loss](https://mlsanity.com/paper/2207.11482/) (2022)

## Abstract (arXiv)

Action recognition via 3D skeleton data is an emerging important topic in these years. Most existing methods either extract hand-crafted descriptors or learn action representations by supervised learning paradigms that require massive labeled data. In this paper, we for the first time propose a contrastive action learning paradigm named AS-CAL that can leverage different augmentations of unlabeled skeleton data to learn action representations in an unsupervised manner. Specifically, we first propose to contrast similarity between augmented instances (query and key) of the input skeleton sequence, which are transformed by multiple novel augmentation strategies, to learn inherent action patterns ("pattern-invariance") of different skeleton transformations. Second, to encourage learning the pattern-invariance with more consistent action representations, we propose a momentum LSTM, which is implemented as the momentum-based moving average of LSTM based query encoder, to encode long-term action dynamics of the key sequence. Third, we introduce a queue to store the encoded keys, which allows our model to flexibly reuse proceeding keys and build a more consistent dictionary to improve contrastive learning. Last, by temporally averaging the hidden states of action learned by the query encoder, a novel representation named Contrastive Action Encoding (CAE) is proposed to represent human's action effectively. Extensive experiments show that our approach typically improves existing hand-crafted methods by 10-50% top-1 accuracy, and it can achieve comparable or even superior performance to numerous supervised learning methods.