# Investigating Pose Representations and Motion Contexts Modeling for 3D Motion Prediction

arXiv: https://arxiv.org/abs/2112.15012 · HTML: https://mlsanity.com/paper/2112.15012/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 56 references that are in the map.
- First posted on arXiv 30 Dec 2021.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Unsupervised Learning of Video Representations using LSTMs](https://mlsanity.com/paper/1502.04681/) (2015)
- [On human motion prediction using recurrent neural networks](https://mlsanity.com/paper/1705.02445/) (2017)
- [Learning Trajectory Dependencies for Human Motion Prediction](https://mlsanity.com/paper/1908.05436/) (2019)
- [Structural-RNN: Deep Learning on Spatio-Temporal Graphs](https://mlsanity.com/paper/1511.05298/) (2015)
- [Human Motion Trajectory Prediction: A Survey](https://mlsanity.com/paper/1905.06113/) (2019)
- [A Spatio-temporal Transformer for 3D Human Motion Prediction](https://mlsanity.com/paper/2004.08692/) (2020)
- [Dynamic Multiscale Graph Neural Networks for 3D Skeleton-Based Human Motion Prediction](https://mlsanity.com/paper/2003.08802/) (2020)
- [Convolutional Sequence to Sequence Model for Human Dynamics](https://mlsanity.com/paper/1805.00655/) (2018)
- [Peeking into the Future: Predicting Future Person Activities and Locations in Videos](https://mlsanity.com/paper/1902.03748/) (2019)
- [Every Pixel Counts ++: Joint Learning of Geometry and Motion with 3D Holistic Understanding](https://mlsanity.com/paper/1810.06125/) (2018)
- [Deep representation learning for human motion prediction and classification](https://mlsanity.com/paper/1702.07486/) (2017)
- [View Adaptive Neural Networks for High Performance Skeleton-based Human Action Recognition](https://mlsanity.com/paper/1804.07453/) (2018)
- [Modeling Human Motion with Quaternion-based Neural Networks](https://mlsanity.com/paper/1901.07677/) (2019)
- [Long-Term Human Motion Prediction by Modeling Motion Context and Enhancing Motion Dynamic](https://mlsanity.com/paper/1805.02513/) (2018)
- [Human Motion Prediction via Learning Local Structure Representations and Temporal Dependencies](https://mlsanity.com/paper/1902.07367/) (2019)
- [Scene-LSTM: A Model for Human Trajectory Prediction](https://mlsanity.com/paper/1808.04018/) (2018)
- [Sentence-State LSTM for Text Representation](https://mlsanity.com/paper/1805.02474/) (2018)
- [A Novel Geometric Framework on Gram Matrix Trajectories for Human Behavior Understanding](https://mlsanity.com/paper/1807.00676/) (2018)
- [Lie-X: Depth Image Based Articulated Object Pose Estimation, Tracking, and Action Recognition on Lie Groups](https://mlsanity.com/paper/1609.03773/) (2016)
- [Trajectory Prediction by Coupling Scene-LSTM with Human Movement LSTM](https://mlsanity.com/paper/1908.08908/) (2019)

## Built on by (most cited first, 30 of 31)

- [MoMask: Generative Masked Modeling of 3D Human Motions](https://mlsanity.com/paper/2312.00063/) (2023)
- [PathAsst: A Generative Foundation AI Assistant Towards Artificial General Intelligence of Pathology](https://mlsanity.com/paper/2305.15072/) (2023)
- [Exploring Optical-Flow-Guided Motion and Detection-Based Appearance for Temporal Sentence Grounding](https://mlsanity.com/paper/2203.02966/) (2022)
- [Diverse Human Motion Prediction via Gumbel-Softmax Sampling from an Auxiliary Space](https://mlsanity.com/paper/2207.07351/) (2022)
- [3D Human Motion Prediction: A Survey](https://mlsanity.com/paper/2203.01593/) (2022)
- [Human Joint Kinematics Diffusion-Refinement for Stochastic Motion Prediction](https://mlsanity.com/paper/2210.05976/) (2022)
- [Scaling Large Motion Models with Million-Level Human Motions](https://mlsanity.com/paper/2410.03311/) (2024)
- [Auxiliary Tasks Benefit 3D Skeleton-based Human Motion Prediction](https://mlsanity.com/paper/2308.08942/) (2023)
- [CgT-GAN: CLIP-guided Text GAN for Image Captioning](https://mlsanity.com/paper/2308.12045/) (2023)
- [Do As I Do: Pose Guided Human Motion Copy](https://mlsanity.com/paper/2406.16601/) (2024)
- [Text-driven Human Motion Generation with Motion Masked Diffusion Model](https://mlsanity.com/paper/2409.19686/) (2024)
- [Context-based Interpretable Spatio-Temporal Graph Convolutional Network for Human Motion Forecasting](https://mlsanity.com/paper/2402.19237/) (2024)
- [Multi-Condition Latent Diffusion Network for Scene-Aware Neural Human Motion Prediction](https://mlsanity.com/paper/2405.18700/) (2024)
- [BiPO: Bidirectional Partial Occlusion Network for Text-to-Motion Synthesis](https://mlsanity.com/paper/2412.00112/) (2024)
- [MotionVerse: A Unified Multimodal Framework for Motion Comprehension, Generation and Editing](https://mlsanity.com/paper/2509.23635/) (2025)
- [REMOT: A Region-to-Whole Framework for Realistic Human Motion Transfer](https://mlsanity.com/paper/2209.00475/) (2022)
- [DMMGAN: Diverse Multi Motion Prediction of 3D Human Joints using Attention-Based Generative Adverserial Network](https://mlsanity.com/paper/2209.09124/) (2022)
- [Mix-up Self-Supervised Learning for Contrast-agnostic Applications](https://mlsanity.com/paper/2204.00901/) (2022)
- [Towards Practical Human Motion Prediction with LiDAR Point Clouds](https://mlsanity.com/paper/2408.08202/) (2024)
- [Motion Diffusion Autoencoders: Enabling Attribute Manipulation in Human Motion Demonstrated on Karate Techniques](https://mlsanity.com/paper/2501.18729/) (2025)
- [HUMOF: Human Motion Forecasting in Interactive Social Scenes](https://mlsanity.com/paper/2506.03753/) (2025)
- [Towards Closed-Loop Embodied Empathy Evolution: Probing LLM-Centric Lifelong Empathic Motion Generation in Unseen Scenarios](https://mlsanity.com/paper/2512.19551/) (2025)
- [Instance-Variant Loss with Gaussian RBF Kernel for 3D Cross-modal Retriveal](https://mlsanity.com/paper/2305.04239/) (2023)
- [ACMo: Attribute Controllable Motion Generation](https://mlsanity.com/paper/2503.11038/) (2025)
- [Personalized Pose Forecasting](https://mlsanity.com/paper/2312.03528/) (2023)
- [RoleMotion: A Large-Scale Dataset towards Robust Scene-Specific Role-Playing Motion Synthesis with Fine-grained Descriptions](https://mlsanity.com/paper/2512.01582/) (2025)
- [PoseMoE: Mixture-of-Experts Network for Monocular 3D Human Pose Estimation](https://mlsanity.com/paper/2512.16494/) (2025)
- [A Self-Supervised Approach on Motion Calibration for Enhancing Physical Plausibility in Text-to-Motion](https://mlsanity.com/paper/2602.18199/) (2026)
- [EgoMotion: Hierarchical Reasoning and Diffusion for Egocentric Vision-Language Motion Generation](https://mlsanity.com/paper/2604.19105/) (2026)
- [From Where to How: Continuous 4D Interaction Forecasting from Egocentric Video](https://mlsanity.com/paper/2609.08636/) (2026)

## Abstract (arXiv)

Predicting human motion from historical pose sequence is crucial for a machine to succeed in intelligent interactions with humans. One aspect that has been obviated so far, is the fact that how we represent the skeletal pose has a critical impact on the prediction results. Yet there is no effort that investigates across different pose representation schemes. We conduct an indepth study on various pose representations with a focus on their effects on the motion prediction task. Moreover, recent approaches build upon off-the-shelf RNN units for motion prediction. These approaches process input pose sequence sequentially and inherently have difficulties in capturing long-term dependencies. In this paper, we propose a novel RNN architecture termed AHMR (Attentive Hierarchical Motion Recurrent network) for motion prediction which simultaneously models local motion contexts and a global context. We further explore a geodesic loss and a forward kinematics loss for the motion prediction task, which have more geometric significance than the widely employed L2 loss. Interestingly, we applied our method to a range of articulate objects including human, fish, and mouse. Empirical results show that our approach outperforms the state-of-the-art methods in short-term prediction and achieves much enhanced long-term prediction proficiency, such as retaining natural human-like motions over 50 seconds predictions. Our codes are released.