mlsanityData to 7 Oct 2026

Hand-Object Interaction Pretraining from Videos

arXiv 2409.08273 ↗First posted 12 Sep 2024cs.RO cs.AI cs.CVcited by 38

Not on the map. Why?

Abstract (arXiv)

We present an approach to learn general robot manipulation priors from 3D hand-object interaction trajectories. We build a framework to use in-the-wild videos to generate sensorimotor robot trajectories. We do so by lifting both the human hand and the manipulated object in a shared 3D space and retargeting human motions to robot actions. Generative modeling on this data gives us a task-agnostic base policy. This policy captures a general yet flexible manipulation prior. We empirically demonstrate that finetuning this policy, with both reinforcement learning (RL) and behavior cloning (BC), enables sample-efficient adaptation to downstream tasks and simultaneously improves robustness and generalizability compared to prior approaches. Qualitative experiments are available at: urlhttps://hgaurav2k.github.io/hop/.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 75 of its 98 references that are in the map. First posted on arXiv 12 Sep 2024.

Builds on

75 of 98 references.

References · 75

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  7. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  9. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  10. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  11. 2019Solving Rubik's Cube with a Robot Handcited by 699
  12. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
38 more
  1. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  2. 2018Learning Dexterous In-Hand Manipulationcited by 643
  3. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  4. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  5. 2024Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robotscited by 373
  6. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 320
  7. 2023Reconstructing Hands in 3D with Transformerscited by 298
  8. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  9. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 256
  10. 2022Masked Visual Pre-training for Motor Controlcited by 246
  11. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  12. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  13. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
  14. 2023Any-point Trajectory Modeling for Policy Learningcited by 207
  15. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 194
  16. 2023GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulatorscited by 180
  17. 2022Human-to-Robot Imitation in the Wildcited by 157
  18. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulationcited by 149
  19. 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 146
  20. 2023Structured World Models from Human Videoscited by 144
  21. 2020Reconstructing Hand-Object Interactions in the Wildcited by 143
  22. 2019H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactionscited by 139
  23. 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 136
  24. 2022OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interactioncited by 129
  25. 2023UniDexGrasp: Universal Robotic Dexterous Grasping via Learning Diverse Proposal Generation and Goal-Conditioned Policycited by 117
  26. 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105
  27. 2022In-Hand Object Rotation via Rapid Motor Adaptationcited by 102
  28. 2022VideoDex: Learning Dexterity from Internet Videoscited by 101
  29. 2022Visual Dexterity: In-Hand Reorientation of Novel and Complex Object Shapescited by 92
  30. 2023PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNavcited by 91
  31. 2022What's in your hands? 3D Reconstruction of Generic Objects in Handscited by 90
  32. 2021Learning by Watching: Physical Imitation of Manipulation Skills from Human Videoscited by 87
  33. 2023General In-Hand Object Rotation with Vision and Touchcited by 87
  34. 2022DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Videocited by 83
  35. 2022Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtubecited by 79
  36. 2022From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperationcited by 79
  37. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  38. 2022Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulationcited by 75

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 28; 2026: 10.

38 papers.

Sort:

By area:

  1. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 643
  2. 2025EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Videocited by 191
  3. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 94
  4. 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videoscited by 63
  5. 2025Visual Imitation Enables Contextual Humanoid Controlcited by 57
  6. 2025DexWild: Dexterous Human Interactions for In-the-Wild Robot Policiescited by 34
  7. 2025ZeroMimic: Distilling Robotic Manipulation Skills from Web Videoscited by 31
  8. 2025EgoZero: Robot Learning from Smart Glassescited by 28
  9. 2025Crossing the Human-Robot Embodiment Gap with Sim-to-Real RL using One Human Demonstrationcited by 27
  10. 2025World Models for Learning Dexterous Hand-Object Interactions from Human Videoscited by 26
  11. 2025Dexterous Manipulation through Imitation Learning: A Surveycited by 18
  12. 2025DexTrack: Towards Generalizable Neural Tracking Control for Dexterous Manipulation from Human Referencescited by 13
26 more
  1. 2025Object-centric 3D Motion Field for Robot Learning from Human Videoscited by 13
  2. 2025MAPLE: Encoding Dexterous Robotic Manipulation Priors Learned From Egocentric Videoscited by 12
  3. 2025SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarioscited by 11
  4. 2025Web2Grasp: Learning Functional Grasps from Web Images of Hand-Object Interactionscited by 10
  5. 2025Spatial-Aware VLA Pretraining through Visual-Physical Alignment from Human Videoscited by 10
  6. 2026Do as I Do: Dexterous Manipulation Data from Everyday Human Videoscited by 9
  7. 2026Dexterous Manipulation Policies from RGB Human Videos via 3D Hand-Object Trajectory Reconstructioncited by 8
  8. 2026Joint-Aligned Latent Action: Towards Scalable VLA Pretraining in the Wildcited by 8
  9. 2025Novel Diffusion Models for Multimodal 3D Hand Trajectory Predictioncited by 7
  10. 2025Robot Learning from Any Imagescited by 7
  11. 2025DexMan: Learning Bimanual Dexterous Manipulation from Human and Generated Videoscited by 7
  12. 2026GeoWorld: Geometric World Modelscited by 6
  13. 2025Emergent Active Perception and Dexterity of Simulated Humanoids from Visual Reinforcement Learningcited by 5
  14. 2026FEEL (Force-Enhanced Egocentric Learning): A Dataset for Physical Action Understandingcited by 4
  15. 2026Glove2Hand: Synthesizing Natural Hand-Object Interaction from Multi-Modal Sensing Glovescited by 3
  16. 2025FunHOI: Annotation-Free 3D Hand-Object Interaction Generation via Functional Text Guidancecited by 2
  17. 2025Learning Adaptive Dexterous Grasping from Single Demonstrationscited by 2
  18. 2025Slot-Level Robotic Placement via Visual Imitation from Single Human Videocited by 2
  19. 2025EmbodiSwap for Zero-Shot Robot Imitation Learningcited by 2
  20. 2025GenDexHand: Generative Simulation for Dexterous Handscited by 1
  21. 2025Mistake Attribution: Fine-Grained Mistake Understanding in Egocentric Videoscited by 1
  22. 2025IGen: Scalable Data Generation for Robot Learning from Open-World Imagescited by 1
  23. 2026Imitating What Works: Simulation-Filtered Modular Policy Learning from Human Videoscited by 1
  24. 2026D-REX: Differentiable Real-to-Sim-to-Real Engine for Learning Dexterous Graspingcited by 1
  25. 2026EgoRecovery: Acquiring Failure Recovery Ability Through Human Recovery Demonstrationcited by 0
  26. 2026HuRo: Robotizing Human Videos for Scalable VLA Pretrainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Learning a Universal Human Prior for Dexterous Manipulation from Human Preference
  2. 2024Dreamitate: Real-World Visuomotor Policy Learning via Video Generation
  3. 2025Learning Generalizable Hand-Object Tracking from Synthetic Demonstrations
  4. 2025TASTE-Rob: Advancing Video Generation of Task-Oriented Hand-Object Interaction for Generalizable Robotic Manipulation
  5. 2026Dexterous Point Policy: Learning Point-based Dexterous Hand Policies from Human Demonstrations
  6. 2026Hand-centric Human-to-Robot Trajectory Transfer from Video Demonstrations via Object Category-agnostic Temporal Localization
  7. 2024Controlling the World by Sleight of Hand
  8. 2024Leveraging Pretrained Latent Representations for Few-Shot Imitation Learning on a Dexterous Robotic Hand