# SkeletonMAE: Graph-based Masked Autoencoder for Skeleton Sequence Pre-training

arXiv: https://arxiv.org/abs/2307.08476 · HTML: https://mlsanity.com/paper/2307.08476/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 76 of its 106 references that are in the map.
- First posted on arXiv 17 Jul 2023.

## Builds on (most cited first, 30 of 76)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [node2vec: Scalable Feature Learning for Networks](https://mlsanity.com/paper/1607.00653/) (2016)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Graph Contrastive Learning with Augmentations](https://mlsanity.com/paper/2010.13902/) (2020)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [LINE: Large-scale Information Network Embedding](https://mlsanity.com/paper/1503.03578/) (2015)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [GraphMAE: Self-Supervised Masked Graph Autoencoders](https://mlsanity.com/paper/2205.10803/) (2022)
- [Unsupervised Learning of Video Representations using LSTMs](https://mlsanity.com/paper/1502.04681/) (2015)
- [Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2003.14111/) (2020)
- [On human motion prediction using recurrent neural networks](https://mlsanity.com/paper/1705.02445/) (2017)
- [Temporal Relational Reasoning in Videos](https://mlsanity.com/paper/1711.08496/) (2017)
- [Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2107.12213/) (2021)

## Built on by (most cited first, 30 of 41)

- [Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AI](https://mlsanity.com/paper/2407.06886/) (2024)
- [Revealing Key Details to See Differences: A Novel Prototypical Perspective for Skeleton-based Action Recognition](https://mlsanity.com/paper/2411.18941/) (2024)
- [Hulk: A Universal Knowledge Translator for Human-Centric Tasks](https://mlsanity.com/paper/2312.01697/) (2023)
- [SignMusketeers: An Efficient Multi-Stream Approach for Sign Language Translation at Scale](https://mlsanity.com/paper/2406.06907/) (2024)
- [USDRL: Unified Skeleton-Based Dense Representation Learning with Multi-Grained Feature Decorrelation](https://mlsanity.com/paper/2412.09220/) (2024)
- [Masked Modeling for Self-supervised Representation Learning on Vision and Beyond](https://mlsanity.com/paper/2401.00897/) (2023)
- [Foundation Model for Skeleton-Based Human Action Understanding](https://mlsanity.com/paper/2508.12586/) (2025)
- [Where to Mask: Structure-Guided Masking for Graph Masked Autoencoders](https://mlsanity.com/paper/2404.15806/) (2024)
- [Idempotent Unsupervised Representation Learning for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2410.20349/) (2024)
- [MEIA: Multimodal Embodied Perception and Interaction in Unknown Environments](https://mlsanity.com/paper/2402.00290/) (2024)
- [Self-Supervised Skeleton-Based Action Representation Learning: A Benchmark and Beyond](https://mlsanity.com/paper/2406.02978/) (2024)
- [Towards Robust and Realistic Human Pose Estimation via WiFi Signals](https://mlsanity.com/paper/2501.09411/) (2025)
- [VideoPure: Diffusion-based Adversarial Purification for Video Recognition](https://mlsanity.com/paper/2501.14999/) (2025)
- [Spatio-Temporal Joint Density Driven Learning for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2505.23012/) (2025)
- [STARS: Self-supervised Tuning for 3D Action Recognition in Skeleton Sequences](https://mlsanity.com/paper/2407.10935/) (2024)
- [Muscles in Time: Learning to Understand Human Motion by Simulating Muscle Activations](https://mlsanity.com/paper/2411.00128/) (2024)
- [iMiGUE-3K: A Large-Scale Benchmark for Micro-Gesture Analysis with Self-Supervised Learning](https://mlsanity.com/paper/2605.17179/) (2026)
- [Skeleton2vec: A Self-supervised Learning Framework with Contextualized Target Representations for Skeleton Sequence](https://mlsanity.com/paper/2401.00921/) (2024)
- [Scalable and Generalizable Correspondence Pruning via Geometry-Consistent Pre-training](https://mlsanity.com/paper/2406.05773/) (2024)
- [Mask and Compress: Efficient Skeleton-based Action Recognition in Continual Learning](https://mlsanity.com/paper/2407.01397/) (2024)
- [Predicting Long-horizon Futures by Conditioning on Geometry and Time](https://mlsanity.com/paper/2404.11554/) (2024)
- [Hi-GMAE: Hierarchical Graph Masked Autoencoders](https://mlsanity.com/paper/2405.10642/) (2024)
- [How do Foundation Models Compare to Skeleton-Based Approaches for Gesture Recognition in Human-Robot Interaction?](https://mlsanity.com/paper/2506.20795/) (2025)
- [SkeletonAgent: An Agentic Interaction Framework for Skeleton-based Action Recognition](https://mlsanity.com/paper/2511.22433/) (2025)
- [Multimodal Skeleton-Based Action Representation Learning via Decomposition and Composition](https://mlsanity.com/paper/2512.21064/) (2025)
- [Lifelong Domain Adaptive 3D Human Pose Estimation](https://mlsanity.com/paper/2512.23860/) (2025)
- [Learning Context-Adaptive Motion Priors for Masked Motion Diffusion Models with Efficient Kinematic Attention Aggregation](https://mlsanity.com/paper/2603.07697/) (2026)
- [AnyMo: Geometry-Aware Setup-Agnostic Modeling of Human Motion in the Wild](https://mlsanity.com/paper/2605.22715/) (2026)
- [Multi-entity Video Transformers for Fine-Grained Video Representation Learning](https://mlsanity.com/paper/2311.10873/) (2023)
- [Spatial Hierarchy and Temporal Attention Guided Cross Masking for Self-supervised Skeleton-based Action Recognition](https://mlsanity.com/paper/2409.17951/) (2024)

## Abstract (arXiv)

Skeleton sequence representation learning has shown great advantages for action recognition due to its promising ability to model human joints and topology. However, the current methods usually require sufficient labeled data for training computationally expensive models, which is labor-intensive and time-consuming. Moreover, these methods ignore how to utilize the fine-grained dependencies among different skeleton joints to pre-train an efficient skeleton sequence learning model that can generalize well across different datasets. In this paper, we propose an efficient skeleton sequence learning framework, named Skeleton Sequence Learning (SSL). To comprehensively capture the human pose and obtain discriminative skeleton sequence representation, we build an asymmetric graph-based encoder-decoder pre-training architecture named SkeletonMAE, which embeds skeleton joint sequence into Graph Convolutional Network (GCN) and reconstructs the masked skeleton joints and edges based on the prior human topology knowledge. Then, the pre-trained SkeletonMAE encoder is integrated with the Spatial-Temporal Representation Learning (STRL) module to build the SSL framework. Extensive experimental results show that our SSL generalizes well across different datasets and outperforms the state-of-the-art self-supervised skeleton-based action recognition methods on FineGym, Diving48, NTU 60 and NTU 120 datasets. Additionally, we obtain comparable performance to some fully supervised methods. The code is avaliable at https://github.com/HongYan1123/SkeletonMAE.