# MotionBERT: A Unified Perspective on Learning Human Motion Representations

arXiv: https://arxiv.org/abs/2210.06551 · HTML: https://mlsanity.com/paper/2210.06551/

## Key facts

- Cited by 193 papers in the map (data through 14 Oct 2026).
- Builds on 98 of its 145 references that are in the map.
- First posted on arXiv 12 Oct 2022.

## Builds on (most cited first, 30 of 98)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image](https://mlsanity.com/paper/1607.08128/) (2016)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision](https://mlsanity.com/paper/1611.09813/) (2016)
- [End-to-End Human Pose and Mesh Reconstruction with Transformers](https://mlsanity.com/paper/2012.09760/) (2020)
- [3D human pose estimation in video with temporal convolutions and semi-supervised training](https://mlsanity.com/paper/1811.11742/) (2018)
- [A simple yet effective baseline for 3d human pose estimation](https://mlsanity.com/paper/1705.03098/) (2017)
- [Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2003.14111/) (2020)
- [PARE: Part Attention Regressor for 3D Human Body Estimation](https://mlsanity.com/paper/2104.08527/) (2021)

## Built on by (most cited first, 30 of 193)

- [WHAM: Reconstructing World-grounded Humans with Accurate 3D Motion](https://mlsanity.com/paper/2312.07531/) (2023)
- [CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character control](https://mlsanity.com/paper/2410.03441/) (2024)
- [EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling](https://mlsanity.com/paper/2401.00374/) (2023)
- [Humanoid Policy ~ Human Policy](https://mlsanity.com/paper/2503.13441/) (2025)
- [MotionAGFormer: Enhancing 3D Human Pose Estimation with a Transformer-GCNFormer Network](https://mlsanity.com/paper/2310.16288/) (2023)
- [AvatarGPT: All-in-One Framework for Motion Understanding, Planning, Generation and Beyond](https://mlsanity.com/paper/2311.16468/) (2023)
- [MAS: Multi-view Ancestral Sampling for 3D motion generation using 2D diffusion](https://mlsanity.com/paper/2310.14729/) (2023)
- [LLMs are Good Action Recognizers](https://mlsanity.com/paper/2404.00532/) (2024)
- [Unimotion: Unifying 3D Human Motion Synthesis and Understanding](https://mlsanity.com/paper/2409.15904/) (2024)
- [Skeleton-in-Context: Unified Skeleton Sequence Modeling with In-Context Learning](https://mlsanity.com/paper/2312.03703/) (2023)
- [A Comprehensive Survey on Human Video Generation: Challenges, Methods, and Insights](https://mlsanity.com/paper/2407.08428/) (2024)
- [3D Human Pose Perception from Egocentric Stereo Videos](https://mlsanity.com/paper/2401.00889/) (2023)
- [FinePOSE: Fine-Grained Prompt-Driven 3D Human Pose Estimation via Diffusion Models](https://mlsanity.com/paper/2405.05216/) (2024)
- [Hulk: A Universal Knowledge Translator for Human-Centric Tasks](https://mlsanity.com/paper/2312.01697/) (2023)
- [Exploring Vision Transformers for 3D Human Motion-Language Models with Motion Patches](https://mlsanity.com/paper/2405.04771/) (2024)
- [RTMW: Real-Time Multi-Person 2D and 3D Whole-body Pose Estimation](https://mlsanity.com/paper/2407.08634/) (2024)
- [HumanDreamer: Generating Controllable Human-Motion Videos via Decoupled Generation](https://mlsanity.com/paper/2503.24026/) (2025)
- [ExpertAF: Expert Actionable Feedback from Video](https://mlsanity.com/paper/2408.00672/) (2024)
- [Sketch2Anim: Towards Transferring Sketch Storyboards into 3D Animation](https://mlsanity.com/paper/2504.19189/) (2025)
- [SAT-HMR: Real-Time Multi-Person 3D Mesh Estimation via Scale-Adaptive Tokens](https://mlsanity.com/paper/2411.19824/) (2024)
- [USDRL: Unified Skeleton-Based Dense Representation Learning with Multi-Grained Feature Decorrelation](https://mlsanity.com/paper/2412.09220/) (2024)
- [Hourglass Tokenizer for Efficient Transformer-Based 3D Human Pose Estimation](https://mlsanity.com/paper/2311.12028/) (2023)
- [MacDiff: Unified Skeleton Modeling with Masked Conditional Diffusion](https://mlsanity.com/paper/2409.10473/) (2024)
- [AthletePose3D: A Benchmark Dataset for 3D Human Pose Estimation and Kinematic Validation in Athletic Movements](https://mlsanity.com/paper/2503.07499/) (2025)
- [Pose Magic: Efficient and Temporally Consistent Human Pose Estimation with a Hybrid Mamba-GCN Network](https://mlsanity.com/paper/2408.02922/) (2024)
- [Heterogeneous Skeleton-Based Action Representation Learning](https://mlsanity.com/paper/2506.03481/) (2025)
- [CrossGLG: LLM Guides One-shot Skeleton-based 3D Action Recognition in a Cross-level Manner](https://mlsanity.com/paper/2403.10082/) (2024)
- [Lifting Motion to the 3D World via 2D Diffusion](https://mlsanity.com/paper/2411.18808/) (2024)
- [Foundation Model for Skeleton-Based Human Action Understanding](https://mlsanity.com/paper/2508.12586/) (2025)
- [Holistic-Motion2D: Scalable Whole-body Human Motion Generation in 2D Space](https://mlsanity.com/paper/2406.11253/) (2024)

## Abstract (arXiv)

We present a unified perspective on tackling various human-centric video tasks by learning human motion representations from large-scale and heterogeneous data resources. Specifically, we propose a pretraining stage in which a motion encoder is trained to recover the underlying 3D motion from noisy partial 2D observations. The motion representations acquired in this way incorporate geometric, kinematic, and physical knowledge about human motion, which can be easily transferred to multiple downstream tasks. We implement the motion encoder with a Dual-stream Spatio-temporal Transformer (DSTformer) neural network. It could capture long-range spatio-temporal relationships among the skeletal joints comprehensively and adaptively, exemplified by the lowest 3D pose estimation error so far when trained from scratch. Furthermore, our proposed framework achieves state-of-the-art performance on all three downstream tasks by simply finetuning the pretrained motion encoder with a simple regression head (1-2 layers), which demonstrates the versatility of the learned motion representations. Code and models are available at https://motionbert.github.io/