# HaLP: Hallucinating Latent Positives for Skeleton-based Self-Supervised Learning of Actions

arXiv: https://arxiv.org/abs/2304.00387 · HTML: https://mlsanity.com/paper/2304.00387/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 64 references that are in the map.
- First posted on arXiv 1 Apr 2023.

## Builds on (most cited first, 30 of 44)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Graph Contrastive Learning with Augmentations](https://mlsanity.com/paper/2010.13902/) (2020)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Contrastive Learning with Hard Negative Samples](https://mlsanity.com/paper/2010.04592/) (2020)
- [A Survey on Contrastive Self-supervised Learning](https://mlsanity.com/paper/2011.00362/) (2020)
- [Hard Negative Mixing for Contrastive Learning](https://mlsanity.com/paper/2010.01028/) (2020)
- [Debiased Contrastive Learning](https://mlsanity.com/paper/2007.00224/) (2020)
- [Revisiting Skeleton-based Action Recognition](https://mlsanity.com/paper/2104.13586/) (2021)
- [Human Action Recognition and Prediction: A Survey](https://mlsanity.com/paper/1806.11230/) (2018)
- [EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition](https://mlsanity.com/paper/1908.08498/) (2019)
- [REPAIR: Removing Representation Bias by Dataset Resampling](https://mlsanity.com/paper/1904.07911/) (2019)
- [3D Human Action Representation Learning via Cross-View Consistency Pursuit](https://mlsanity.com/paper/2104.14466/) (2021)
- [Pymanopt: A Python Toolbox for Optimization on Manifolds using Automatic Differentiation](https://mlsanity.com/paper/1603.03236/) (2016)
- [MS2L: Multi-Task Self-Supervised Learning for Skeleton Based Action Recognition](https://mlsanity.com/paper/2010.05599/) (2020)
- [PREDICT & CLUSTER: Unsupervised Skeleton Based Action Recognition](https://mlsanity.com/paper/1911.12409/) (2019)
- [Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-supervised Action Recognition](https://mlsanity.com/paper/2112.03590/) (2021)
- [Skeleton-Contrastive 3D Action Representation Learning](https://mlsanity.com/paper/2108.03656/) (2021)
- [Un-Mix: Rethinking Image Mixtures for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/2003.05438/) (2020)
- [Augmented Skeleton Based Contrastive Action Learning with Momentum LSTM for Unsupervised Action Recognition](https://mlsanity.com/paper/2008.00188/) (2020)

## Built on by (most cited first, 25 of 25)

- [Confabulation: The Surprising Value of Large Language Model Hallucinations](https://mlsanity.com/paper/2406.04175/) (2024)
- [Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Training](https://mlsanity.com/paper/2305.13689/) (2023)
- [Revealing Key Details to See Differences: A Novel Prototypical Perspective for Skeleton-based Action Recognition](https://mlsanity.com/paper/2411.18941/) (2024)
- [Prompted Contrast with Masked Motion Modeling: Towards Versatile 3D Action Representation Learning](https://mlsanity.com/paper/2308.03975/) (2023)
- [Hallucination Improves the Performance of Unsupervised Visual Representation Learning](https://mlsanity.com/paper/2307.12168/) (2023)
- [USDRL: Unified Skeleton-Based Dense Representation Learning with Multi-Grained Feature Decorrelation](https://mlsanity.com/paper/2412.09220/) (2024)
- [Heterogeneous Skeleton-Based Action Representation Learning](https://mlsanity.com/paper/2506.03481/) (2025)
- [Vision-Language Meets the Skeleton: Progressively Distillation with Cross-Modal Knowledge for 3D Action Representation Learning](https://mlsanity.com/paper/2405.20606/) (2024)
- [Foundation Model for Skeleton-Based Human Action Understanding](https://mlsanity.com/paper/2508.12586/) (2025)
- [No Free Lunch in Self Supervised Representation Learning](https://mlsanity.com/paper/2304.11718/) (2023)
- [Human Pose-based Estimation, Tracking and Action Recognition with Deep Learning: A Survey](https://mlsanity.com/paper/2310.13039/) (2023)
- [MV2MAE: Multi-View Video Masked Autoencoders](https://mlsanity.com/paper/2401.15900/) (2024)
- [3D Skeleton-Based Action Recognition: A Review](https://mlsanity.com/paper/2506.00915/) (2025)
- [Self-Supervised Skeleton-Based Action Representation Learning: A Benchmark and Beyond](https://mlsanity.com/paper/2406.02978/) (2024)
- [LightHouse: A Survey of AGI Hallucination](https://mlsanity.com/paper/2401.06792/) (2024)
- [STARS: Self-supervised Tuning for 3D Action Recognition in Skeleton Sequences](https://mlsanity.com/paper/2407.10935/) (2024)
- [Towards Efficient General Feature Prediction in Masked Skeleton Modeling](https://mlsanity.com/paper/2509.03609/) (2025)
- [Skeleton2vec: A Self-supervised Learning Framework with Contextualized Target Representations for Skeleton Sequence](https://mlsanity.com/paper/2401.00921/) (2024)
- [DuoCLR: Dual-Surrogate Contrastive Learning for Skeleton-based Human Action Segmentation](https://mlsanity.com/paper/2509.05543/) (2025)
- [Attack-Augmentation Mixing-Contrastive Skeletal Representation Learning](https://mlsanity.com/paper/2304.04023/) (2023)
- [Multimodal Skeleton-Based Action Representation Learning via Decomposition and Composition](https://mlsanity.com/paper/2512.21064/) (2025)
- [Unsupervised Spatial-Temporal Feature Enrichment and Fidelity Preservation Network for Skeleton based Action Recognition](https://mlsanity.com/paper/2401.14034/) (2024)
- [Spatial Hierarchy and Temporal Attention Guided Cross Masking for Self-supervised Skeleton-based Action Recognition](https://mlsanity.com/paper/2409.17951/) (2024)
- [Beyond Binary Contrast: Modeling Continuous Skeleton Action Spaces with Transitional Anchors](https://mlsanity.com/paper/2604.17914/) (2026)
- [One for All: Generalist Foundation Model for Cross-Sensor Skeleton Representation Learning](https://mlsanity.com/paper/2609.07078/) (2026)

## Abstract (arXiv)

Supervised learning of skeleton sequence encoders for action recognition has received significant attention in recent times. However, learning such encoders without labels continues to be a challenging problem. While prior works have shown promising results by applying contrastive learning to pose sequences, the quality of the learned representations is often observed to be closely tied to data augmentations that are used to craft the positives. However, augmenting pose sequences is a difficult task as the geometric constraints among the skeleton joints need to be enforced to make the augmentations realistic for that action. In this work, we propose a new contrastive learning approach to train models for skeleton-based action recognition without labels. Our key contribution is a simple module, HaLP - to Hallucinate Latent Positives for contrastive learning. Specifically, HaLP explores the latent space of poses in suitable directions to generate new positives. To this end, we present a novel optimization formulation to solve for the synthetic positives with an explicit control on their hardness. We propose approximations to the objective, making them solvable in closed form with minimal overhead. We show via experiments that using these generated positives within a standard contrastive learning framework leads to consistent improvements across benchmarks such as NTU-60, NTU-120, and PKU-II on tasks like linear evaluation, transfer learning, and kNN evaluation. Our code will be made available at https://github.com/anshulbshah/HaLP.