# Towards Efficient General Feature Prediction in Masked Skeleton Modeling

arXiv: https://arxiv.org/abs/2509.03609 · HTML: https://mlsanity.com/paper/2509.03609/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 44 references that are in the map.
- First posted on arXiv 3 Sep 2025.

## Builds on (most cited first, 30 of 33)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [Temporal Action Detection with Structured Segment Networks](https://mlsanity.com/paper/1704.06228/) (2017)
- [3D Human Action Representation Learning via Cross-View Consistency Pursuit](https://mlsanity.com/paper/2104.14466/) (2021)
- [PREDICT & CLUSTER: Unsupervised Skeleton Based Action Recognition](https://mlsanity.com/paper/1911.12409/) (2019)
- [Skeleton-Contrastive 3D Action Representation Learning](https://mlsanity.com/paper/2108.03656/) (2021)
- [Augmented Skeleton Based Contrastive Action Learning with Momentum LSTM for Unsupervised Action Recognition](https://mlsanity.com/paper/2008.00188/) (2020)
- [Skeleton Cloud Colorization for Unsupervised 3D Action Representation Learning](https://mlsanity.com/paper/2108.01959/) (2021)
- [TubeR: Tubelet Transformer for Video Action Detection](https://mlsanity.com/paper/2104.00969/) (2021)
- [HYperbolic Self-Paced Learning for Self-Supervised Skeleton-based Action Representations](https://mlsanity.com/paper/2303.06242/) (2023)
- [Actionlet-Dependent Contrastive Learning for Unsupervised Skeleton-Based Action Recognition](https://mlsanity.com/paper/2303.10904/) (2023)
- [Masked Motion Predictors are Strong 3D Action Representation Learners](https://mlsanity.com/paper/2308.07092/) (2023)
- [CMD: Self-supervised 3D Action Representation Learning with Cross-modal Mutual Distillation](https://mlsanity.com/paper/2208.12448/) (2022)
- [Adversarial Self-Supervised Learning for Semi-Supervised 3D Action Recognition](https://mlsanity.com/paper/2007.05934/) (2020)
- [Contrastive Positive Mining for Unsupervised 3D Action Representation Learning](https://mlsanity.com/paper/2208.03497/) (2022)
- [Global-local Motion Transformer for Unsupervised Skeleton-based Action Learning](https://mlsanity.com/paper/2207.06101/) (2022)
- [Unsupervised 3D Human Pose Representation with Viewpoint and Pose Disentanglement](https://mlsanity.com/paper/2007.07053/) (2020)
- [Hierarchically Self-Supervised Transformer for Human Skeleton Representation Learning](https://mlsanity.com/paper/2207.09644/) (2022)
- [Hierarchical Contrast for Unsupervised Skeleton-based Action Representation Learning](https://mlsanity.com/paper/2212.02082/) (2022)
- [SkeletonMAE: Spatial-Temporal Masked Autoencoders for Self-supervised Skeleton Action Recognition](https://mlsanity.com/paper/2209.02399/) (2022)
- [Unsupervised Feature Learning of Human Actions as Trajectories in Pose Embedding Manifold](https://mlsanity.com/paper/1812.02592/) (2018)
- [EulerMormer: Robust Eulerian Motion Magnification via Dynamic Filtering within Transformer](https://mlsanity.com/paper/2312.04152/) (2023)
- [HaLP: Hallucinating Latent Positives for Skeleton-based Self-Supervised Learning of Actions](https://mlsanity.com/paper/2304.00387/) (2023)
- [ViA: View-invariant Skeleton Action Representation Learning via Motion Retargeting](https://mlsanity.com/paper/2209.00065/) (2022)
- [Part Aware Contrastive Learning for Self-Supervised Action Recognition](https://mlsanity.com/paper/2305.00666/) (2023)
- [Unified Multi-modal Unsupervised Representation Learning for Skeleton-based Action Understanding](https://mlsanity.com/paper/2311.03106/) (2023)
- [Prompted Contrast with Masked Motion Modeling: Towards Versatile 3D Action Representation Learning](https://mlsanity.com/paper/2308.03975/) (2023)

## Built on by (most cited first, 3 of 3)

- [Beyond Binary Contrast: Modeling Continuous Skeleton Action Spaces with Transitional Anchors](https://mlsanity.com/paper/2604.17914/) (2026)
- [Action Motifs: Self-Supervised Hierarchical Representation of Human Body Movements](https://mlsanity.com/paper/2604.28173/) (2026)
- [One for All: Generalist Foundation Model for Cross-Sensor Skeleton Representation Learning](https://mlsanity.com/paper/2609.07078/) (2026)

## Abstract (arXiv)

Recent advances in the masked autoencoder (MAE) paradigm have significantly propelled self-supervised skeleton-based action recognition. However, most existing approaches limit reconstruction targets to raw joint coordinates or their simple variants, resulting in computational redundancy and limited semantic representation. To address this, we propose a novel General Feature Prediction framework (GFP) for efficient mask skeleton modeling. Our key innovation is replacing conventional low-level reconstruction with high-level feature prediction that spans from local motion patterns to global semantic representations. Specifically, we introduce a collaborative learning framework where a lightweight target generation network dynamically produces diversified supervision signals across spatial-temporal hierarchies, avoiding reliance on pre-computed offline features. The framework incorporates constrained optimization to ensure feature diversity while preventing model collapse. Experiments on NTU RGB+D 60, NTU RGB+D 120 and PKU-MMD demonstrate the benefits of our approach: Computational efficiency (with 6.2× faster training than standard masked skeleton modeling methods) and superior representation quality, achieving state-of-the-art performance in various downstream tasks.