# Auxiliary Tasks Benefit 3D Skeleton-based Human Motion Prediction

arXiv: https://arxiv.org/abs/2308.08942 · HTML: https://mlsanity.com/paper/2308.08942/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 78 references that are in the map.
- First posted on arXiv 17 Aug 2023.

## Builds on (most cited first, 30 of 51)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [Simple and Deep Graph Convolutional Networks](https://mlsanity.com/paper/2007.02133/) (2020)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling](https://mlsanity.com/paper/2111.14819/) (2021)
- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [Masked Feature Prediction for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2112.09133/) (2021)
- [Masked Autoencoders for Point Cloud Self-supervised Learning](https://mlsanity.com/paper/2203.06604/) (2022)
- [Masked Autoencoders As Spatiotemporal Learners](https://mlsanity.com/paper/2205.09113/) (2022)
- [Reinforcement Learning with Unsupervised Auxiliary Tasks](https://mlsanity.com/paper/1611.05397/) (2016)
- [Learning Lane Graph Representations for Motion Forecasting](https://mlsanity.com/paper/2007.13732/) (2020)
- [On human motion prediction using recurrent neural networks](https://mlsanity.com/paper/1705.02445/) (2017)
- [MultiMAE: Multi-modal Multi-task Masked Autoencoders](https://mlsanity.com/paper/2204.01678/) (2022)
- [Learning Trajectory Dependencies for Human Motion Prediction](https://mlsanity.com/paper/1908.05436/) (2019)
- [Structural-RNN: Deep Learning on Spatio-Temporal Graphs](https://mlsanity.com/paper/1511.05298/) (2015)
- [Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction](https://mlsanity.com/paper/1611.09842/) (2016)
- [DeepStereo: Learning to Predict New Views from the World's Imagery](https://mlsanity.com/paper/1506.06825/) (2015)
- [History Repeats Itself: Human Motion Prediction via Motion Attention](https://mlsanity.com/paper/2007.11755/) (2020)
- [A Spatio-temporal Transformer for 3D Human Motion Prediction](https://mlsanity.com/paper/2004.08692/) (2020)
- [Dynamic Multiscale Graph Neural Networks for 3D Skeleton-Based Human Motion Prediction](https://mlsanity.com/paper/2003.08802/) (2020)
- [Distilling Knowledge from Graph Convolutional Networks](https://mlsanity.com/paper/2003.10477/) (2020)
- [GroupNet: Multiscale Hypergraph Neural Networks for Trajectory Prediction with Relational Reasoning](https://mlsanity.com/paper/2204.08770/) (2022)

## Built on by (most cited first, 20 of 20)

- [Multi-agent Long-term 3D Human Pose Forecasting via Interaction-aware Trajectory Conditioning](https://mlsanity.com/paper/2404.05218/) (2024)
- [Semantic-CC: Boosting Remote Sensing Image Change Captioning via Foundational Knowledge and Semantic Guidance](https://mlsanity.com/paper/2407.14032/) (2024)
- [Motion Generation: A Survey of Generative Approaches and Benchmarks](https://mlsanity.com/paper/2507.05419/) (2025)
- [Multimodal Sense-Informed Prediction of 3D Human Motions](https://mlsanity.com/paper/2405.02911/) (2024)
- [RealTraj: Towards Real-World Pedestrian Trajectory Forecasting](https://mlsanity.com/paper/2411.17376/) (2024)
- [Compatible Transformer for Irregularly Sampled Multivariate Time Series](https://mlsanity.com/paper/2310.11022/) (2023)
- [HUMOF: Human Motion Forecasting in Interactive Social Scenes](https://mlsanity.com/paper/2506.03753/) (2025)
- [MMHU: A Massive-Scale Multimodal Benchmark for Human Behavior Understanding](https://mlsanity.com/paper/2507.12463/) (2025)
- [Recent Advances in Deterministic Human Motion Prediction: A Review](https://mlsanity.com/paper/2312.06184/) (2023)
- [Off-the-shelf ChatGPT is a Good Few-shot Human Motion Predictor](https://mlsanity.com/paper/2405.15267/) (2024)
- [Spatio-Temporal Multi-Subgraph GCN for 3D Human Motion Prediction](https://mlsanity.com/paper/2501.00317/) (2024)
- [Enhancing Spatio-Temporal Zero-shot Action Recognition with Language-driven Description Attributes](https://mlsanity.com/paper/2510.27255/) (2025)
- [Temporal Dynamics Decoupling with Inverse Processing for Enhancing Human Motion Prediction](https://mlsanity.com/paper/2501.00315/) (2024)
- [Enhancing Human Motion Prediction via Multi-range Decoupling Decoding with Gating-adjusting Aggregation](https://mlsanity.com/paper/2503.23381/) (2025)
- [GGMotion: Group Graph Dynamics-Kinematics Networks for Human Motion Prediction](https://mlsanity.com/paper/2507.07515/) (2025)
- [Scriboora: Rethinking Human Pose Forecasting](https://mlsanity.com/paper/2511.15565/) (2025)
- [JOintGS: Joint Optimization of Cameras, Bodies and 3D Gaussians for In-the-Wild Monocular Reconstruction](https://mlsanity.com/paper/2602.04317/) (2026)
- [SIGMAS: Second-Order Interaction-based Grouping for Overlapping Multi-Agent Swarms](https://mlsanity.com/paper/2603.00120/) (2026)
- [KOALA: Koopman Operator Learning for WiFi-Based Anticipatory Hum](https://mlsanity.com/paper/2608.15815/) (2026)
- [SPARC: Single-Pass Scaling for Motion Forecasting with Conformal Bayesian Last Layers](https://mlsanity.com/paper/2608.20802/) (2026)

## Abstract (arXiv)

Exploring spatial-temporal dependencies from observed motions is one of the core challenges of human motion prediction. Previous methods mainly focus on dedicated network structures to model the spatial and temporal dependencies. This paper considers a new direction by introducing a model learning framework with auxiliary tasks. In our auxiliary tasks, partial body joints' coordinates are corrupted by either masking or adding noise and the goal is to recover corrupted coordinates depending on the rest coordinates. To work with auxiliary tasks, we propose a novel auxiliary-adapted transformer, which can handle incomplete, corrupted motion data and achieve coordinate recovery via capturing spatial-temporal dependencies. Through auxiliary tasks, the auxiliary-adapted transformer is promoted to capture more comprehensive spatial-temporal dependencies among body joints' coordinates, leading to better feature learning. Extensive experimental results have shown that our method outperforms state-of-the-art methods by remarkable margins of 7.2%, 3.7%, and 9.4% in terms of 3D mean per joint position error (MPJPE) on the Human3.6M, CMU Mocap, and 3DPW datasets, respectively. We also demonstrate that our method is more robust under data missing cases and noisy data cases. Code is available at https://github.com/MediaBrain-SJTU/AuxFormer.