# UNIK: A Unified Framework for Real-world Skeleton-based Action Recognition

arXiv: https://arxiv.org/abs/2107.08580 · HTML: https://mlsanity.com/paper/2107.08580/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 57 references that are in the map.
- First posted on arXiv 19 Jul 2021.

## Builds on (most cited first, 30 of 44)

- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [3D human pose estimation in video with temporal convolutions and semi-supervised training](https://mlsanity.com/paper/1811.11742/) (2018)
- [Convolutional Two-Stream Network Fusion for Video Action Recognition](https://mlsanity.com/paper/1604.06573/) (2016)
- [Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2003.14111/) (2020)
- [RMPE: Regional Multi-person Pose Estimation](https://mlsanity.com/paper/1612.00137/) (2016)
- [Everybody Dance Now](https://mlsanity.com/paper/1808.07371/) (2018)
- [A Short Note on the Kinetics-700 Human Action Dataset](https://mlsanity.com/paper/1907.06987/) (2019)
- [Video Classification with Channel-Separated Convolutional Networks](https://mlsanity.com/paper/1904.02811/) (2019)
- [Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognition](https://mlsanity.com/paper/1904.12659/) (2019)
- [TDN: Temporal Difference Networks for Efficient Action Recognition](https://mlsanity.com/paper/2012.10071/) (2020)
- [Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition](https://mlsanity.com/paper/1708.07632/) (2017)
- [Cross-view Action Modeling, Learning and Recognition](https://mlsanity.com/paper/1405.2941/) (2014)
- [An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data](https://mlsanity.com/paper/1611.06067/) (2016)
- [Co-occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregation](https://mlsanity.com/paper/1804.06055/) (2018)
- [3D Human Action Representation Learning via Cross-View Consistency Pursuit](https://mlsanity.com/paper/2104.14466/) (2021)
- [Skeleton-Based Action Recognition with Multi-Stream Adaptive Graph Convolutional Networks](https://mlsanity.com/paper/1912.06971/) (2019)
- [UAV-Human: A Large Benchmark for Human Behavior Understanding with Unmanned Aerial Vehicles](https://mlsanity.com/paper/2104.00946/) (2021)
- [Stronger, Faster and More Explainable: A Graph Convolutional Baseline for Skeleton-based Action Recognition](https://mlsanity.com/paper/2010.09978/) (2020)
- [Interpretable 3D Human Action Analysis with Temporal Convolutional Networks](https://mlsanity.com/paper/1704.04516/) (2017)
- [LCR-Net++: Multi-person 2D and 3D Pose Detection in Natural Images](https://mlsanity.com/paper/1803.00455/) (2018)

## Built on by (most cited first, 30 of 35)

- [MotionBERT: A Unified Perspective on Learning Human Motion Representations](https://mlsanity.com/paper/2210.06551/) (2022)
- [STAR-Transformer: A Spatio-temporal Cross Attention Transformer for Human Action Recognition](https://mlsanity.com/paper/2210.07503/) (2022)
- [LLMs are Good Action Recognizers](https://mlsanity.com/paper/2404.00532/) (2024)
- [PhysPT: Physics-aware Pretrained Transformer for Estimating Human Dynamics from Monocular Videos](https://mlsanity.com/paper/2404.04430/) (2024)
- [DG-STGCN: Dynamic Spatial-Temporal Modeling for Skeleton-based Action Recognition](https://mlsanity.com/paper/2210.05895/) (2022)
- [ViA: View-invariant Skeleton Action Representation Learning via Motion Retargeting](https://mlsanity.com/paper/2209.00065/) (2022)
- [Hulk: A Universal Knowledge Translator for Human-Centric Tasks](https://mlsanity.com/paper/2312.01697/) (2023)
- [ViewCLR: Learning Self-supervised Video Representation for Unseen Viewpoints](https://mlsanity.com/paper/2112.03905/) (2021)
- [Cross-Modal Learning with 3D Deformable Attention for Action Recognition](https://mlsanity.com/paper/2212.05638/) (2022)
- [Focal and Global Spatial-Temporal Transformer for Skeleton-based Action Recognition](https://mlsanity.com/paper/2210.02693/) (2022)
- [Self-Supervised 3D Action Representation Learning with Skeleton Cloud Colorization](https://mlsanity.com/paper/2304.08799/) (2023)
- [LAC: Latent Action Composition for Skeleton-based Action Segmentation](https://mlsanity.com/paper/2308.14500/) (2023)
- [Simba: Mamba augmented U-ShiftGCN for Skeletal Action Recognition in Videos](https://mlsanity.com/paper/2404.07645/) (2024)
- [Self-Supervised Video Representation Learning via Latent Time Navigation](https://mlsanity.com/paper/2305.06437/) (2023)
- [HumMUSS: Human Motion Understanding using State Space Models](https://mlsanity.com/paper/2404.10880/) (2024)
- [MoFM: A Large-Scale Human Motion Foundation Model](https://mlsanity.com/paper/2502.05432/) (2025)
- [Towards To-a-T Spatio-Temporal Focus for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2202.02314/) (2022)
- [Collaborating Domain-shared and Target-specific Feature Clustering for Cross-domain 3D Action Recognition](https://mlsanity.com/paper/2207.09767/) (2022)
- [Are Visual-Language Models Effective in Action Recognition? A Comparative Study](https://mlsanity.com/paper/2410.17149/) (2024)
- [Transformers in Action Recognition: A Review on Temporal Modeling](https://mlsanity.com/paper/2302.01921/) (2022)
- [Just Dance with π! A Poly-modal Inductor for Weakly-supervised Video Anomaly Detection](https://mlsanity.com/paper/2505.13123/) (2025)
- [Feature Hallucination for Self-supervised Action Recognition](https://mlsanity.com/paper/2506.20342/) (2025)
- [Cross-Model Cross-Stream Learning for Self-Supervised Human Action Recognition](https://mlsanity.com/paper/2307.07791/) (2023)
- [DuoCLR: Dual-Surrogate Contrastive Learning for Skeleton-based Human Action Segmentation](https://mlsanity.com/paper/2509.05543/) (2025)
- [Topic Detection in Continuous Sign Language Videos](https://mlsanity.com/paper/2209.02402/) (2022)
- [SkeleTR: Towrads Skeleton-based Action Recognition in the Wild](https://mlsanity.com/paper/2309.11445/) (2023)
- [AM Flow: Adapters for Temporal Processing in Action Recognition](https://mlsanity.com/paper/2411.02065/) (2024)
- [PoseBH: Prototypical Multi-Dataset Training Beyond Human Pose Estimation](https://mlsanity.com/paper/2505.17475/) (2025)
- [Towards Generalizing Temporal Action Segmentation to Unseen Views](https://mlsanity.com/paper/2504.02512/) (2025)
- [Skeleton-Snippet Contrastive Learning with Multiscale Feature Fusion for Action Localization](https://mlsanity.com/paper/2512.16504/) (2025)

## Abstract (arXiv)

Action recognition based on skeleton data has recently witnessed increasing attention and progress. State-of-the-art approaches adopting Graph Convolutional networks (GCNs) can effectively extract features on human skeletons relying on the pre-defined human topology. Despite associated progress, GCN-based methods have difficulties to generalize across domains, especially with different human topological structures. In this context, we introduce UNIK, a novel skeleton-based action recognition method that is not only effective to learn spatio-temporal features on human skeleton sequences but also able to generalize across datasets. This is achieved by learning an optimal dependency matrix from the uniform distribution based on a multi-head attention mechanism. Subsequently, to study the cross-domain generalizability of skeleton-based action recognition in real-world videos, we re-evaluate state-of-the-art approaches as well as the proposed UNIK in light of a novel Posetics dataset. This dataset is created from Kinetics-400 videos by estimating, refining and filtering poses. We provide an analysis on how much performance improves on smaller benchmark datasets after pre-training on Posetics for the action classification task. Experimental results show that the proposed UNIK, with pre-training on Posetics, generalizes well and outperforms state-of-the-art when transferred onto four target action classification datasets: Toyota Smarthome, Penn Action, NTU-RGB+D 60 and NTU-RGB+D 120.