# Unsupervised Feature Learning of Human Actions as Trajectories in Pose Embedding Manifold

arXiv: https://arxiv.org/abs/1812.02592 · HTML: https://mlsanity.com/paper/1812.02592/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Dec 2018.

## Built on by (most cited first, 27 of 27)

- [3D Human Action Representation Learning via Cross-View Consistency Pursuit](https://mlsanity.com/paper/2104.14466/) (2021)
- [Skeleton Cloud Colorization for Unsupervised 3D Action Representation Learning](https://mlsanity.com/paper/2108.01959/) (2021)
- [Contrastive Positive Mining for Unsupervised 3D Action Representation Learning](https://mlsanity.com/paper/2208.03497/) (2022)
- [Global-local Motion Transformer for Unsupervised Skeleton-based Action Learning](https://mlsanity.com/paper/2207.06101/) (2022)
- [Hierarchically Self-Supervised Transformer for Human Skeleton Representation Learning](https://mlsanity.com/paper/2207.09644/) (2022)
- [Hierarchical Contrast for Unsupervised Skeleton-based Action Representation Learning](https://mlsanity.com/paper/2212.02082/) (2022)
- [Generative Adversarial Graph Convolutional Networks for Human Action Synthesis](https://mlsanity.com/paper/2110.11191/) (2021)
- [ViA: View-invariant Skeleton Action Representation Learning via Motion Retargeting](https://mlsanity.com/paper/2209.00065/) (2022)
- [Spatiotemporal Decouple-and-Squeeze Contrastive Learning for Semi-Supervised Skeleton-based Action Recognition](https://mlsanity.com/paper/2302.02316/) (2023)
- [Self-Supervised 3D Action Representation Learning with Skeleton Cloud Colorization](https://mlsanity.com/paper/2304.08799/) (2023)
- [Pyramid Self-attention Polymerization Learning for Semi-supervised Skeleton-based Action Recognition](https://mlsanity.com/paper/2302.02327/) (2023)
- [SCD-Net: Spatiotemporal Clues Disentanglement Network for Self-supervised Skeleton-based Action Recognition](https://mlsanity.com/paper/2309.05834/) (2023)
- [Unsupervised Human Action Recognition with Skeletal Graph Laplacian and Self-Supervised Viewpoints Invariance](https://mlsanity.com/paper/2204.10312/) (2022)
- [Bootstrapped Representation Learning for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2202.02232/) (2022)
- [Non-Local Latent Relation Distillation for Self-Adaptive 3D Human Pose Estimation](https://mlsanity.com/paper/2204.01971/) (2022)
- [DMMG: Dual Min-Max Games for Self-Supervised Skeleton-Based Action Recognition](https://mlsanity.com/paper/2302.12007/) (2023)
- [Focalized Contrastive View-invariant Learning for Self-supervised Skeleton-based Action Recognition](https://mlsanity.com/paper/2304.00858/) (2023)
- [Collaborating Domain-shared and Target-specific Feature Clustering for Cross-domain 3D Action Recognition](https://mlsanity.com/paper/2207.09767/) (2022)
- [Token Boosting for Robust Self-Supervised Visual Transformer Pre-training](https://mlsanity.com/paper/2304.04175/) (2023)
- [Dynamic Object Removal for Effective Slam](https://mlsanity.com/paper/2303.10923/) (2023)
- [Towards Efficient General Feature Prediction in Masked Skeleton Modeling](https://mlsanity.com/paper/2509.03609/) (2025)
- [Learning video embedding space with Natural Language Supervision](https://mlsanity.com/paper/2303.14584/) (2023)
- [Attack-Augmentation Mixing-Contrastive Skeletal Representation Learning](https://mlsanity.com/paper/2304.04023/) (2023)
- [ReL-SAR: Representation Learning for Skeleton Action Recognition with Convolutional Transformers and BYOL](https://mlsanity.com/paper/2409.05749/) (2024)
- [TransDocs: Optical Character Recognition with word to word translation](https://mlsanity.com/paper/2304.07637/) (2023)
- [Matching Skeleton-based Activity Representations with Heterogeneous Signals for HAR](https://mlsanity.com/paper/2503.14547/) (2025)
- [QCLR: A Plug-in Robustification Module via Quantization for Contrastive Learning in Self-Supervised Skeleton-Based Human Action Recognition](https://mlsanity.com/paper/2610.13281/) (2026)

## Abstract (arXiv)

An unsupervised human action modeling framework can provide useful pose-sequence representation, which can be utilized in a variety of pose analysis applications. In this work we propose a novel temporal pose-sequence modeling framework, which can embed the dynamics of 3D human-skeleton joints to a continuous latent space in an efficient manner. In contrast to end-to-end framework explored by previous works, we disentangle the task of individual pose representation learning from the task of learning actions as a trajectory in pose embedding space. In order to realize a continuous pose embedding manifold with improved reconstructions, we propose an unsupervised, manifold learning procedure named Encoder GAN, (or EnGAN). Further, we use the pose embeddings generated by EnGAN to model human actions using a bidirectional RNN auto-encoder architecture, PoseRNN. We introduce first-order gradient loss to explicitly enforce temporal regularity in the predicted motion sequence. A hierarchical feature fusion technique is also investigated for simultaneous modeling of local skeleton joints along with global pose variations. We demonstrate state-of-the-art transfer-ability of the learned representation against other supervisedly and unsupervisedly learned motion embeddings for the task of fine-grained action recognition on SBU interaction dataset. Further, we show the qualitative strengths of the proposed framework by visualizing skeleton pose reconstructions and interpolations in pose-embedding space, and low dimensional principal component projections of the reconstructed pose trajectories.