# Unsupervised Learning of View-invariant Action Representations

arXiv: https://arxiv.org/abs/1809.01844 · HTML: https://mlsanity.com/paper/1809.01844/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Sep 2018.

## Built on by (most cited first, 30 of 32)

- [3D Human Action Representation Learning via Cross-View Consistency Pursuit](https://mlsanity.com/paper/2104.14466/) (2021)
- [PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences](https://mlsanity.com/paper/2205.13713/) (2022)
- [iMiGUE: An Identity-free Video Dataset for Micro-Gesture Understanding and Emotion Analysis](https://mlsanity.com/paper/2107.00285/) (2021)
- [Skeleton Cloud Colorization for Unsupervised 3D Action Representation Learning](https://mlsanity.com/paper/2108.01959/) (2021)
- [Learning Cross-modal Contrastive Features for Video Domain Adaptation](https://mlsanity.com/paper/2108.11974/) (2021)
- [VARS: Video Assistant Referee System for Automated Soccer Decision Making from Multiple Views](https://mlsanity.com/paper/2304.04617/) (2023)
- [No Pain, Big Gain: Classify Dynamic Point Cloud Sequences with Static Models by Fitting Feature-level Space-time Surfaces](https://mlsanity.com/paper/2203.11113/) (2022)
- [ViA: View-invariant Skeleton Action Representation Learning via Motion Retargeting](https://mlsanity.com/paper/2209.00065/) (2022)
- [ViewCLR: Learning Self-supervised Video Representation for Unseen Viewpoints](https://mlsanity.com/paper/2112.03905/) (2021)
- [DVANet: Disentangling View and Action Features for Multi-View Action Recognition](https://mlsanity.com/paper/2312.05719/) (2023)
- [Spatiotemporal Decouple-and-Squeeze Contrastive Learning for Semi-Supervised Skeleton-based Action Recognition](https://mlsanity.com/paper/2302.02316/) (2023)
- [Unsupervised Motion Representation Learning with Capsule Autoencoders](https://mlsanity.com/paper/2110.00529/) (2021)
- [Self-Supervised 3D Action Representation Learning with Skeleton Cloud Colorization](https://mlsanity.com/paper/2304.08799/) (2023)
- [Unsupervised Human Action Recognition with Skeletal Graph Laplacian and Self-Supervised Viewpoints Invariance](https://mlsanity.com/paper/2204.10312/) (2022)
- [Identity-free Artificial Emotional Intelligence via Micro-Gesture Understanding](https://mlsanity.com/paper/2405.13206/) (2024)
- [Focalized Contrastive View-invariant Learning for Self-supervised Skeleton-based Action Recognition](https://mlsanity.com/paper/2304.00858/) (2023)
- [MV2MAE: Multi-View Video Masked Autoencoders](https://mlsanity.com/paper/2401.15900/) (2024)
- [PALMAR: Towards Adaptive Multi-inhabitant Activity Recognition in Point-Cloud Technology](https://mlsanity.com/paper/2106.11902/) (2021)
- [Self-Supervised Skeleton-Based Action Representation Learning: A Benchmark and Beyond](https://mlsanity.com/paper/2406.02978/) (2024)
- [Unsupervised View-Invariant Human Posture Representation](https://mlsanity.com/paper/2109.08730/) (2021)
- [Human-in-Context: Unified Cross-Domain 3D Human Motion Modeling via In-Context Learning](https://mlsanity.com/paper/2508.10897/) (2025)
- [Action Recognition with Domain Invariant Features of Skeleton Image](https://mlsanity.com/paper/2111.11250/) (2021)
- [Towards Privacy-Supporting Fall Detection via Deep Unsupervised RGB2Depth Adaptation](https://mlsanity.com/paper/2308.12049/) (2023)
- [Dual-view Spatio-Temporal Feature Fusion with CNN-Transformer Hybrid Network for Chinese Isolated Sign Language Recognition](https://mlsanity.com/paper/2506.06966/) (2025)
- [View-Invariant Skeleton-based Action Recognition via Global-Local Contrastive Learning](https://mlsanity.com/paper/2209.11634/) (2022)
- [Cross-view Action Recognition via Contrastive View-invariant Representation](https://mlsanity.com/paper/2305.01733/) (2023)
- [Object-based (yet Class-agnostic) Video Domain Adaptation](https://mlsanity.com/paper/2311.17942/) (2023)
- [Adversarially Masked Video Consistency for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2403.16242/) (2024)
- [Towards Generalizing Temporal Action Segmentation to Unseen Views](https://mlsanity.com/paper/2504.02512/) (2025)
- [ViewBridge: Curriculum Knowledge Distillation for Activity View-Invariance Under Extreme Viewpoint Changes](https://mlsanity.com/paper/2504.05451/) (2025)

## Abstract (arXiv)

The recent success in human action recognition with deep learning methods mostly adopt the supervised learning paradigm, which requires significant amount of manually labeled data to achieve good performance. However, label collection is an expensive and time-consuming process. In this work, we propose an unsupervised learning framework, which exploits unlabeled data to learn video representations. Different from previous works in video representation learning, our unsupervised learning task is to predict 3D motion in multiple target views using video representation from a source view. By learning to extrapolate cross-view motions, the representation can capture view-invariant motion dynamics which is discriminative for the action. In addition, we propose a view-adversarial training method to enhance learning of view-invariant features. We demonstrate the effectiveness of the learned representations for action recognition on multiple datasets.