# HMP: Hand Motion Priors for Pose and Shape Estimation from Video

arXiv: https://arxiv.org/abs/2312.16737 · HTML: https://mlsanity.com/paper/2312.16737/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 60 of its 72 references that are in the map.
- First posted on arXiv 27 Dec 2023.

## Builds on (most cited first, 30 of 60)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [Action-Conditioned 3D Human Motion Synthesis with Transformer VAE](https://mlsanity.com/paper/2104.05670/) (2021)
- [End-to-End Human Pose and Mesh Reconstruction with Transformers](https://mlsanity.com/paper/2012.09760/) (2020)
- [GRAB: A Dataset of Whole-Body Human Grasping of Objects](https://mlsanity.com/paper/2008.11200/) (2020)
- [Learning joint reconstruction of hands and manipulated objects](https://mlsanity.com/paper/1904.05767/) (2019)
- [PhysDiff: Physics-Guided Human Motion Diffusion Model](https://mlsanity.com/paper/2212.02500/) (2022)
- [On human motion prediction using recurrent neural networks](https://mlsanity.com/paper/1705.02445/) (2017)
- [DexYCB: A Benchmark for Capturing Hand Grasping of Objects](https://mlsanity.com/paper/2104.04631/) (2021)
- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation](https://mlsanity.com/paper/2204.13662/) (2022)
- [Diffusion-based Generation, Optimization, and Planning in 3D Scenes](https://mlsanity.com/paper/2301.06015/) (2023)
- [Structural-RNN: Deep Learning on Spatio-Temporal Graphs](https://mlsanity.com/paper/1511.05298/) (2015)
- [Learning to Estimate 3D Hand Pose from Single RGB Images](https://mlsanity.com/paper/1705.01389/) (2017)
- [Robust Motion In-betweening](https://mlsanity.com/paper/2102.04942/) (2021)
- [Stochastic Scene-Aware Motion Prediction](https://mlsanity.com/paper/2108.08284/) (2021)
- [Long-term Human Motion Prediction with Scene Context](https://mlsanity.com/paper/2007.03672/) (2020)
- [3D Hand Shape and Pose from Images in the Wild](https://mlsanity.com/paper/1902.03451/) (2019)
- [DLow: Diversifying Latent Flows for Diverse Human Motion Prediction](https://mlsanity.com/paper/2003.08386/) (2020)
- [Embodied Hands: Modeling and Capturing Hands and Bodies Together](https://mlsanity.com/paper/2201.02610/) (2022)
- [HP-GAN: Probabilistic 3D human motion prediction via GAN](https://mlsanity.com/paper/1711.09561/) (2017)
- [PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images](https://mlsanity.com/paper/2207.06400/) (2022)
- [GANerated Hands for Real-time 3D Hand Tracking from Monocular RGB](https://mlsanity.com/paper/1712.01057/) (2017)

## Built on by (most cited first, 23 of 23)

- [HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videos](https://mlsanity.com/paper/2501.02973/) (2025)
- [GraspXL: Generating Grasping Motions for Diverse Objects at Scale](https://mlsanity.com/paper/2403.19649/) (2024)
- [Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Camera](https://mlsanity.com/paper/2412.12861/) (2024)
- [Hamba: Single-view 3D Hand Reconstruction with Graph-guided Bi-Scanning Mamba](https://mlsanity.com/paper/2407.09646/) (2024)
- [Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objects](https://mlsanity.com/paper/2403.16428/) (2024)
- [BimArt: A Unified Approach for the Synthesis of 3D Bimanual Interaction with Articulated Objects](https://mlsanity.com/paper/2412.05066/) (2024)
- [Predicting 4D Hand Trajectory from Monocular Videos](https://mlsanity.com/paper/2501.08329/) (2025)
- [MagicHOI: Leveraging 3D Priors for Accurate Hand-object Reconstruction from Short Monocular Video Clips](https://mlsanity.com/paper/2508.05506/) (2025)
- [Survey on Hand Gesture Recognition from Visual Input](https://mlsanity.com/paper/2501.11992/) (2025)
- [MEgoHand: Multimodal Egocentric Hand-Object Interaction Motion Generation](https://mlsanity.com/paper/2505.16602/) (2025)
- [MaskHand: Generative Masked Modeling for Robust Hand Mesh Reconstruction in the Wild](https://mlsanity.com/paper/2412.13393/) (2024)
- [Diffusion-based 3D Hand Motion Recovery with Intuitive Physics](https://mlsanity.com/paper/2508.01835/) (2025)
- [The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction](https://mlsanity.com/paper/2606.30308/) (2026)
- [Towards Dynamic 3D Reconstruction of Hand-Instrument Interaction in Ophthalmic Surgery](https://mlsanity.com/paper/2505.17677/) (2025)
- [PALM: A Dataset and Baseline for Learning Multi-subject Hand Prior](https://mlsanity.com/paper/2511.05403/) (2025)
- [UniHand: A Unified Model for Diverse Controlled 4D Hand Motion Modeling](https://mlsanity.com/paper/2602.21631/) (2026)
- [StableHand: Quality-Aware Flow Matching for World-Space Dual-Hand Motion Estimation from Egocentric Video](https://mlsanity.com/paper/2605.18553/) (2026)
- [HandFlow: Fully Generative 4D Hand Recovery with Flow Matching](https://mlsanity.com/paper/2607.11221/) (2026)
- [MINT: A Unified Model for World-Space Camera and Hand Motion Estimation from Scalable Egocentric Pipeline Supervision](https://mlsanity.com/paper/2609.04958/) (2026)
- [FUSION: Full-Body Unified Motion Prior for Body and Hands via Diffusion](https://mlsanity.com/paper/2601.03959/) (2026)
- [REACH: Hand Pose Estimation from Room Corners](https://mlsanity.com/paper/2605.22231/) (2026)
- [Hand-4DGS: Feed-Forward 3D Gaussian Splatting for 4D Hand Reconstruction from Egocentric Videos](https://mlsanity.com/paper/2606.19156/) (2026)
- [ESTHER: Egocentric Stereo Hand Estimation and Reconstruction in the Wild](https://mlsanity.com/paper/2609.34817/) (2026)

## Abstract (arXiv)

Understanding how humans interact with the world necessitates accurate 3D hand pose estimation, a task complicated by the hand's high degree of articulation, frequent occlusions, self-occlusions, and rapid motions. While most existing methods rely on single-image inputs, videos have useful cues to address aforementioned issues. However, existing video-based 3D hand datasets are insufficient for training feedforward models to generalize to in-the-wild scenarios. On the other hand, we have access to large human motion capture datasets which also include hand motions, e.g. AMASS. Therefore, we develop a generative motion prior specific for hands, trained on the AMASS dataset which features diverse and high-quality hand motions. This motion prior is then employed for video-based 3D hand motion estimation following a latent optimization approach. Our integration of a robust motion prior significantly enhances performance, especially in occluded scenarios. It produces stable, temporally consistent results that surpass conventional single-frame methods. We demonstrate our method's efficacy via qualitative and quantitative evaluations on the HO3D and DexYCB datasets, with special emphasis on an occlusion-focused subset of HO3D. Code is available at https://hmp.is.tue.mpg.de