# V2P-Manip: Learning Dexterous Manipulation from Monocular Human Videos

arXiv: https://arxiv.org/abs/2606.16436 · HTML: https://mlsanity.com/paper/2606.16436/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 60 of its 78 references that are in the map.
- First posted on arXiv 15 Jun 2026.

## Builds on (most cited first, 30 of 60)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects](https://mlsanity.com/paper/2312.08344/) (2023)
- [End-to-End Human Pose and Mesh Reconstruction with Transformers](https://mlsanity.com/paper/2012.09760/) (2020)
- [GRAB: A Dataset of Whole-Body Human Grasping of Objects](https://mlsanity.com/paper/2008.11200/) (2020)
- [Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation](https://mlsanity.com/paper/2401.02117/) (2024)
- [UniDepth: Universal Monocular Metric Depth Estimation](https://mlsanity.com/paper/2403.18913/) (2024)
- [3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions](https://mlsanity.com/paper/1603.08182/) (2016)
- [Deep Residual Learning in Spiking Neural Networks](https://mlsanity.com/paper/2102.04159/) (2021)
- [MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations](https://mlsanity.com/paper/2310.17596/) (2023)
- [DexYCB: A Benchmark for Capturing Hand Grasping of Objects](https://mlsanity.com/paper/2104.04631/) (2021)
- [Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism](https://mlsanity.com/paper/2103.12021/) (2021)
- [Residual Reinforcement Learning for Robot Control](https://mlsanity.com/paper/1812.03201/) (2018)
- [MoGe-2: Accurate Monocular Geometry with Metric Scale and Sharp Details](https://mlsanity.com/paper/2507.02546/) (2025)
- [SAM 3D: 3Dfy Anything in Images](https://mlsanity.com/paper/2511.16624/) (2025)
- [DexMV: Imitation Learning for Dexterous Manipulation from Human Videos](https://mlsanity.com/paper/2108.05877/) (2021)
- [SpatialTracker: Tracking Any 2D Pixels in 3D Space](https://mlsanity.com/paper/2404.04319/) (2024)
- [DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation](https://mlsanity.com/paper/2403.07788/) (2024)
- [AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System](https://mlsanity.com/paper/2307.04577/) (2023)
- [H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions](https://mlsanity.com/paper/1904.05349/) (2019)
- [ContactOpt: Optimizing Contact to Improve Grasps](https://mlsanity.com/paper/2104.07267/) (2021)
- [DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on Simulation](https://mlsanity.com/paper/2210.02697/) (2022)
- [OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interaction](https://mlsanity.com/paper/2203.15709/) (2022)
- [WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wild](https://mlsanity.com/paper/2409.12259/) (2024)
- [Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning](https://mlsanity.com/paper/2206.08686/) (2022)
- [DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning](https://mlsanity.com/paper/2410.24185/) (2024)

## Abstract (arXiv)

Achieving autonomous robotic dexterous manipulation requires precise, human-like action sequences at scale. As a scalable supplement to costly teleoperation data, extracting trajectories with both visual fidelity and physical plausibility from monocular videos represents a promising frontier in embodied AI. To this end, we introduce V2P-Manip, an efficient framework designed to learn dexterous manipulation policies directly from human demonstration videos. We establish an efficient, integrated pipeline encompassing 3D asset acquisition, trajectory estimation, and dexterous policy learning. To bridge the gap between visual perception and physical constraints, we introduce a two-stage refinement process to enforce spatial alignment and physical consistency. Evaluations on the TACO and OakInk benchmarks demonstrate that our approach significantly outperforms previous methods in pose accuracy, adaptability to unstructured environments, and training efficiency. Ultimately, experimental results confirm an average success rate of over 75% across multiple synthetic manipulation tasks and validate the adaptability of the extracted manipulation priors across diverse dexterous hand embodiments.