# Object-Centric Dexterous Manipulation from Human Motion Data

arXiv: https://arxiv.org/abs/2411.04005 · HTML: https://mlsanity.com/paper/2411.04005/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Nov 2024.

## Built on by (most cited first, 30 of 41)

- [DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Grasping](https://mlsanity.com/paper/2502.20900/) (2025)
- [ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning](https://mlsanity.com/paper/2503.21860/) (2025)
- [RoboVerse: Towards a Unified Platform, Dataset and Benchmark for Scalable and Generalizable Robot Learning](https://mlsanity.com/paper/2504.18904/) (2025)
- [Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids](https://mlsanity.com/paper/2502.20396/) (2025)
- [DexMachina: Functional Retargeting for Bimanual Dexterous Manipulation](https://mlsanity.com/paper/2505.24853/) (2025)
- [Crossing the Human-Robot Embodiment Gap with Sim-to-Real RL using One Human Demonstration](https://mlsanity.com/paper/2504.12609/) (2025)
- [SkillMimic-V2: Learning Robust and Generalizable Interaction Skills from Sparse and Noisy Demonstrations](https://mlsanity.com/paper/2505.02094/) (2025)
- [Dex1B: Learning with 1B Demonstrations for Dexterous Manipulation](https://mlsanity.com/paper/2506.17198/) (2025)
- [DemoDiffusion: One-Shot Human Imitation using pre-trained Diffusion Policy](https://mlsanity.com/paper/2506.20668/) (2025)
- [SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation](https://mlsanity.com/paper/2602.16863/) (2026)
- [DexMan: Learning Bimanual Dexterous Manipulation from Human and Generated Videos](https://mlsanity.com/paper/2510.08475/) (2025)
- [Scaffolding Dexterous Manipulation with Vision-Language Models](https://mlsanity.com/paper/2506.19212/) (2025)
- [GenFlowRL: Shaping Rewards with Generative Object-Centric Flow in Visual Reinforcement Learning](https://mlsanity.com/paper/2508.11049/) (2025)
- [From Simple to Complex Skills: The Case of In-Hand Object Reorientation](https://mlsanity.com/paper/2501.05439/) (2025)
- [Novel Diffusion Models for Multimodal 3D Hand Trajectory Prediction](https://mlsanity.com/paper/2504.07375/) (2025)
- [RUKA: Rethinking the Design of Humanoid Hands with Learning](https://mlsanity.com/paper/2504.13165/) (2025)
- [Robot Trains Robot: Automatic Real-World Policy Adaptation and Learning for Humanoids](https://mlsanity.com/paper/2508.12252/) (2025)
- [Robot Learning from Any Images](https://mlsanity.com/paper/2509.22970/) (2025)
- [TopoRetarget: Interaction-Preserving Retargeting for Dexterous Manipulation](https://mlsanity.com/paper/2606.16272/) (2026)
- [Learning Dexterous Manipulation Using Contact Wrench Guidance From Human Demonstration](https://mlsanity.com/paper/2607.00033/) (2026)
- [LodeStar: Long-horizon Dexterity via Synthetic Data Augmentation from Human Demonstrations](https://mlsanity.com/paper/2508.17547/) (2025)
- [FAR-Dex: Few-shot Data Augmentation and Adaptive Residual Policy Refinement for Dexterous Manipulation](https://mlsanity.com/paper/2603.10451/) (2026)
- [Learning Long-Horizon Robot Manipulation Skills via Privileged Action](https://mlsanity.com/paper/2502.15442/) (2025)
- [Learning Adaptive Dexterous Grasping from Single Demonstrations](https://mlsanity.com/paper/2503.20208/) (2025)
- [Dex2HOI: Dexterous Bimanual Two-Object Interaction Generation](https://mlsanity.com/paper/2605.30444/) (2026)
- [DexCanvas: Bridging Human Demonstrations and Robot Learning for Dexterous Manipulation](https://mlsanity.com/paper/2510.15786/) (2025)
- [IGen: Scalable Data Generation for Robot Learning from Open-World Images](https://mlsanity.com/paper/2512.01773/) (2025)
- [UniBYD: A Unified Framework for Learning Robotic Manipulation Across Embodiments Beyond Imitation of Human Demonstrations](https://mlsanity.com/paper/2512.11609/) (2025)
- [Hand-Object Interaction in the Age of Large Foundation Models:Reconstruction, Generation, and Embodied Transfer](https://mlsanity.com/paper/2607.28394/) (2026)
- [AsymDex: Asymmetry and Relative Coordinates for RL-based Bimanual Dexterity](https://mlsanity.com/paper/2411.13020/) (2024)

## Abstract (arXiv)

Manipulating objects to achieve desired goal states is a basic but important skill for dexterous manipulation. Human hand motions demonstrate proficient manipulation capability, providing valuable data for training robots with multi-finger hands. Despite this potential, substantial challenges arise due to the embodiment gap between human and robot hands. In this work, we introduce a hierarchical policy learning framework that uses human hand motion data for training object-centric dexterous robot manipulation. At the core of our method is a high-level trajectory generative model, learned with a large-scale human hand motion capture dataset, to synthesize human-like wrist motions conditioned on the desired object goal states. Guided by the generated wrist motions, deep reinforcement learning is further used to train a low-level finger controller that is grounded in the robot's embodiment to physically interact with the object to achieve the goal. Through extensive evaluation across 10 household objects, our approach not only demonstrates superior performance but also showcases generalization capability to novel object geometries and goal states. Furthermore, we transfer the learned policies from simulation to a real-world bimanual dexterous robot system, further demonstrating its applicability in real-world scenarios. Project website: https://cypypccpy.github.io/obj-dex.github.io/.