mlsanityData to 7 Oct 2026

MotionBERT: A Unified Perspective on Learning Human Motion Representations

arXiv 2210.06551 ↗First posted 12 Oct 2022cs.CVcited by 188

Not on the map. Why?

Abstract (arXiv)

We present a unified perspective on tackling various human-centric video tasks by learning human motion representations from large-scale and heterogeneous data resources. Specifically, we propose a pretraining stage in which a motion encoder is trained to recover the underlying 3D motion from noisy partial 2D observations. The motion representations acquired in this way incorporate geometric, kinematic, and physical knowledge about human motion, which can be easily transferred to multiple downstream tasks. We implement the motion encoder with a Dual-stream Spatio-temporal Transformer (DSTformer) neural network. It could capture long-range spatio-temporal relationships among the skeletal joints comprehensively and adaptively, exemplified by the lowest 3D pose estimation error so far when trained from scratch. Furthermore, our proposed framework achieves state-of-the-art performance on all three downstream tasks by simply finetuning the pretrained motion encoder with a simple regression head (1-2 layers), which demonstrates the versatility of the learned motion representations. Code and models are available at https://motionbert.github.io/

Key facts. Cited by 188 papers in the map (data through 7 Oct 2026). Builds on 98 of its 145 references that are in the map. First posted on arXiv 12 Oct 2022.

Builds on

98 of 145 references.

References · 98

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  10. 2020Supervised Contrastive Learningcited by 2,803
  11. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  12. 2019Deep High-Resolution Representation Learning for Human Pose Estimationcited by 1,339
38 more
  1. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  2. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,199
  3. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  4. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,061
  5. 2016Stacked Hourglass Networks for Human Pose Estimationcited by 913
  6. 2017End-to-end Recovery of Human Shape and Posecited by 883
  7. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
  8. 2016NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysiscited by 683
  9. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 625
  10. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
  11. 2018DensePose: Dense Human Pose Estimation In The Wildcited by 573
  12. 2019NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understandingcited by 530
  13. 2016Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervisioncited by 493
  14. 2020End-to-End Human Pose and Mesh Reconstruction with Transformerscited by 417
  15. 2017A simple yet effective baseline for 3d human pose estimationcited by 401
  16. 20183D human pose estimation in video with temporal convolutions and semi-supervised trainingcited by 401
  17. 2020Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognitioncited by 339
  18. 2021PARE: Part Attention Regressor for 3D Human Body Estimationcited by 334
  19. 2021Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognitioncited by 317
  20. 2018Learning 3D Human Dynamics from Videocited by 315
  21. 2020HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimationcited by 304
  22. 2020I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Imagecited by 274
  23. 20213D Human Pose Estimation with Spatial and Temporal Transformerscited by 271
  24. 2019Convolutional Mesh Regression for Single-Image Human Shape Reconstructioncited by 270
  25. 2021Mesh Graphormercited by 264
  26. 2021Revisiting Skeleton-based Action Recognitioncited by 256
  27. 2017Integral Human Pose Regressioncited by 255
  28. 2021PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loopcited by 250
  29. 2020Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Posecited by 248
  30. 2022CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimationcited by 237
  31. 2020FineGym: A Hierarchical Video Dataset for Fine-grained Action Understandingcited by 232
  32. 2019Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognitioncited by 228
  33. 2020Monocular, One-stage, Regression of Multiple 3D Peoplecited by 216
  34. 2017PoseTrack: A Benchmark for Human Pose Estimation and Trackingcited by 173
  35. 2019Distribution-Aware Coordinate Representation for Human Pose Estimationcited by 165
  36. 2020Beyond Static Features for Temporally Consistent 3D Human Pose and Shape from a Videocited by 162
  37. 2022MixSTE: Seq2seq Mixed Spatio-Temporal Encoder for 3D Human Pose Estimation in Videocited by 158
  38. 2021Probabilistic Modeling for Human Mesh Recoverycited by 152

50 most cited of 98 All 98 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 44; 2025: 67; 2026: 65.

188 papers.

Sort:

By area:

  1. 2023WHAM: Reconstructing World-grounded Humans with Accurate 3D Motioncited by 200
  2. 2024CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character controlcited by 103
  3. 2023EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modelingcited by 101
  4. 2025Humanoid Policy ~ Human Policycited by 65
  5. 2023AvatarGPT: All-in-One Framework for Motion Understanding, Planning, Generation and Beyondcited by 63
  6. 2023MotionAGFormer: Enhancing 3D Human Pose Estimation with a Transformer-GCNFormer Networkcited by 62
  7. 2023MAS: Multi-view Ancestral Sampling for 3D motion generation using 2D diffusioncited by 37
  8. 2024LLMs are Good Action Recognizerscited by 36
  9. 2024Unimotion: Unifying 3D Human Motion Synthesis and Understandingcited by 29
  10. 2023Skeleton-in-Context: Unified Skeleton Sequence Modeling with In-Context Learningcited by 26
  11. 2024A Comprehensive Survey on Human Video Generation: Challenges, Methods, and Insightscited by 24
  12. 20233D Human Pose Perception from Egocentric Stereo Videoscited by 23
38 more
  1. 2024FinePOSE: Fine-Grained Prompt-Driven 3D Human Pose Estimation via Diffusion Modelscited by 23
  2. 2023Hulk: A Universal Knowledge Translator for Human-Centric Taskscited by 22
  3. 2024Exploring Vision Transformers for 3D Human Motion-Language Models with Motion Patchescited by 22
  4. 2025HumanDreamer: Generating Controllable Human-Motion Videos via Decoupled Generationcited by 19
  5. 2024RTMW: Real-Time Multi-Person 2D and 3D Whole-body Pose Estimationcited by 18
  6. 2024ExpertAF: Expert Actionable Feedback from Videocited by 17
  7. 2025Sketch2Anim: Towards Transferring Sketch Storyboards into 3D Animationcited by 16
  8. 2023Hourglass Tokenizer for Efficient Transformer-Based 3D Human Pose Estimationcited by 15
  9. 2024MacDiff: Unified Skeleton Modeling with Masked Conditional Diffusioncited by 15
  10. 2024SAT-HMR: Real-Time Multi-Person 3D Mesh Estimation via Scale-Adaptive Tokenscited by 15
  11. 2024USDRL: Unified Skeleton-Based Dense Representation Learning with Multi-Grained Feature Decorrelationcited by 15
  12. 2025AthletePose3D: A Benchmark Dataset for 3D Human Pose Estimation and Kinematic Validation in Athletic Movementscited by 14
  13. 2024Pose Magic: Efficient and Temporally Consistent Human Pose Estimation with a Hybrid Mamba-GCN Networkcited by 12
  14. 2025Heterogeneous Skeleton-Based Action Representation Learningcited by 12
  15. 2024CrossGLG: LLM Guides One-shot Skeleton-based 3D Action Recognition in a Cross-level Mannercited by 11
  16. 2024Lifting Motion to the 3D World via 2D Diffusioncited by 11
  17. 2025Foundation Model for Skeleton-Based Human Action Understandingcited by 11
  18. 2024Holistic-Motion2D: Scalable Whole-body Human Motion Generation in 2D Spacecited by 10
  19. 2024Motion-2-To-3: Leveraging 2D Motion Data for 3D Motion Generationscited by 9
  20. 2025FinePhys: Fine-grained Human Action Generation by Explicitly Incorporating Physical Laws for Effective Skeletal Guidancecited by 9
  21. 2025H-MoRe: Learning Human-centric Motion Representation for Action Analysiscited by 8
  22. 2025Text-driven Motion Generation: Overview, Challenges and Directionscited by 8
  23. 2025Motion Generation: A Survey of Generative Approaches and Benchmarkscited by 8
  24. 2024MotionMap: Representing Multimodality in Human Pose Forecastingcited by 7
  25. 2025Bring Your Rear Cameras for Egocentric 3D Human Pose Estimationcited by 7
  26. 2025RadarLLM: Empowering Large Language Models to Understand Human Motion from Millimeter-Wave Point Cloud Sequencecited by 7
  27. 2024HumMUSS: Human Motion Understanding using State Space Modelscited by 6
  28. 2024UPose3D: Uncertainty-Aware 3D Human Pose Estimation with Cross-View and Temporal Cuescited by 6
  29. 2024Benchmarking Skeleton-based Motion Encoder Models for Clinical Applications: Estimating Parkinson's Disease Severity in Walking Sequencescited by 6
  30. 2024Optimal-state Dynamics Estimation for Physics-based Human Motion Capture from Videoscited by 6
  31. 2025DiSRT-In-Bed: Diffusion-Based Sim-to-Real Transfer Framework for In-Bed Human Mesh Recoverycited by 6
  32. 2026Driver-WM: A Driver-Centric Traffic-Conditioned Latent World Model for In-Cabin Dynamics Rolloutcited by 6
  33. 2024Self-Supervised Skeleton-Based Action Representation Learning: A Benchmark and Beyondcited by 5
  34. 2025TT3D: Table Tennis 3D Reconstructioncited by 5
  35. 2025H2OT: Hierarchical Hourglass Tokenizer for Efficient Video Pose Transformerscited by 5
  36. 2025CARE-PD: A Multi-Site Anonymized Clinical Dataset for Parkinson's Disease Gait Assessmentcited by 5
  37. 2024HDBN: A Novel Hybrid Dual-branch Network for Robust Skeleton-based Action Recognitioncited by 4
  38. 2024Motion Capture from Inertial and Vision Sensorscited by 4

50 most cited of 188 All 188 (JSON)