# The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction

arXiv: https://arxiv.org/abs/2606.30308 · HTML: https://mlsanity.com/paper/2606.30308/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 29 of its 33 references that are in the map.
- First posted on arXiv 29 Jun 2026.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)

## Builds on (most cited first, 29 of 29)

- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [VACE: All-in-One Video Creation and Editing](https://mlsanity.com/paper/2503.07598/) (2025)
- [Emergent Correspondence from Image Diffusion](https://mlsanity.com/paper/2306.03881/) (2023)
- [HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction](https://mlsanity.com/paper/2203.01577/) (2022)
- [Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation](https://mlsanity.com/paper/2312.02145/) (2023)
- [ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation](https://mlsanity.com/paper/2204.13662/) (2022)
- [Reconstructing Hands in 3D with Transformers](https://mlsanity.com/paper/2312.05251/) (2023)
- [EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video](https://mlsanity.com/paper/2505.11709/) (2025)
- [EgoMimic: Scaling Imitation Learning via Egocentric Video](https://mlsanity.com/paper/2410.24221/) (2024)
- [HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos](https://mlsanity.com/paper/2411.19167/) (2024)
- [EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videos](https://mlsanity.com/paper/2507.12440/) (2025)
- [WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wild](https://mlsanity.com/paper/2409.12259/) (2024)
- [EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data](https://mlsanity.com/paper/2602.16710/) (2026)
- [HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videos](https://mlsanity.com/paper/2501.02973/) (2025)
- [Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Camera](https://mlsanity.com/paper/2412.12861/) (2024)
- [Hamba: Single-view 3D Hand Reconstruction with Graph-guided Bi-Scanning Mamba](https://mlsanity.com/paper/2407.09646/) (2024)
- [Bringing Inputs to Shared Domains for 3D Interacting Hands Recovery in the Wild](https://mlsanity.com/paper/2303.13652/) (2023)
- [3D Hand Pose Estimation in Everyday Egocentric Images](https://mlsanity.com/paper/2312.06583/) (2023)
- [Emergent Temporal Correspondences from Video Diffusion Transformers](https://mlsanity.com/paper/2506.17220/) (2025)
- [HMP: Hand Motion Priors for Pose and Shape Estimation from Video](https://mlsanity.com/paper/2312.16737/) (2023)
- [Deformer: Dynamic Fusion Transformer for Robust Hand Pose Estimation](https://mlsanity.com/paper/2303.04991/) (2023)
- [Predicting 4D Hand Trajectory from Monocular Videos](https://mlsanity.com/paper/2501.08329/) (2025)
- [Hand2World: Autoregressive Egocentric Interaction Generation via Free-Space Hand Gestures](https://mlsanity.com/paper/2602.09600/) (2026)
- [From Image to Video: An Empirical Study of Diffusion Representations](https://mlsanity.com/paper/2502.07001/) (2025)
- [OmniHands: Towards Robust 4D Hand Mesh Recovery via A Versatile Transformer](https://mlsanity.com/paper/2405.20330/) (2024)
- [Denoise to Track: Harnessing Video Diffusion Priors for Robust Correspondence](https://mlsanity.com/paper/2512.04619/) (2025)

## Built on by (most cited first, 5 of 5)

- [ACE-Ego-Hand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Motion Recovery](https://mlsanity.com/paper/2608.20308/) (2026)
- [MINT: A Unified Model for World-Space Camera and Hand Motion Estimation from Scalable Egocentric Pipeline Supervision](https://mlsanity.com/paper/2609.04958/) (2026)
- [ESTHER: Egocentric Stereo Hand Estimation and Reconstruction in the Wild](https://mlsanity.com/paper/2609.34817/) (2026)
- [InfiniHand: Streaming World-Space Hand Motion Estimation from Egocentric Video](https://mlsanity.com/paper/2609.35743/) (2026)
- [RLHND: Video Foundation Models as Physically Grounded Hand Trackers for Robot Learning](https://mlsanity.com/paper/2610.09455/) (2026)

## Abstract (arXiv)

4D hand motion reconstruction from egocentric video is bottlenecked by clear limitations of existing methods: image-based pipelines depend on a detector that fails under heavy occlusion, while video-based methods rely on temporal modules learned only from scarce hand-pose annotations, a narrow signal insufficient to model motion dynamics, occlusion reasoning, and hand-object interaction. These capabilities, however, are exactly what video generative models must implicitly acquire when trained to synthesize coherent video at internet scale. Motivated by this, we present ViDiHand, which leverages the representations of a pretrained video diffusion model to reconstruct 4D two-hand pose. We adapt it via a hand-overlay rendering objective that specializes its features for hands while preserving its world priors. A decoder then recovers metric-scale pose from the adapted features. The whole pipeline operates directly on full frames--no detector, no infiller, and no test-time optimization. On ARCTIC, HOT3D, and HOI4D, ViDiHand substantially outperforms prior methods, establishing video diffusion models as a powerful new foundation for hand motion reconstruction and a promising route to scalable in-the-wild data collection for embodied AI. Project page: https://vidihand.github.io.