mlsanityData to 7 Oct 2026

Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Plans

arXiv 2312.00775 ↗First posted 1 Dec 2023cs.RO cs.CV cs.LGcited by 51

Not on the map. Why?

Abstract (arXiv)

We pursue the goal of developing robots that can interact zero-shot with generic unseen objects via a diverse repertoire of manipulation skills and show how passive human videos can serve as a rich source of data for learning such generalist robots. Unlike typical robot learning approaches which directly learn how a robot should act from interaction data, we adopt a factorized approach that can leverage large-scale human videos to learn how a human would accomplish a desired task (a human plan), followed by translating this plan to the robots embodiment. Specifically, we learn a human plan predictor that, given a current image of a scene and a goal image, predicts the future hand and object configurations. We combine this with a translation module that learns a plan-conditioned robot manipulation policy, and allows following humans plans for generic manipulation tasks in a zero-shot manner with no deployment-time training. Importantly, while the plan predictor can leverage large-scale human videos for learning, the translation module only requires a small amount of in-domain data, and can generalize to tasks not seen during training. We show that our learned system can perform over 16 manipulation skills that generalize to 40 objects, encompassing 100 real-world tasks for table-top manipulation and diverse in-the-wild manipulation. https://homangab.github.io/hopman/

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Builds on 56 of its 64 references that are in the map. First posted on arXiv 1 Dec 2023.

Builds on

56 of 64 references.

References · 56

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  3. 2017The Kinetics Human Action Video Datasetcited by 1,894
  4. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  5. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  6. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  7. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  8. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 768
  9. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  10. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  11. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  12. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 307
38 more
  1. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  2. 2022Masked Visual Pre-training for Motor Controlcited by 246
  3. 2017SSD-6D: Making RGB-based 3D detection and 6D pose estimation great againcited by 243
  4. 2017Learning to Estimate 3D Hand Pose from Single RGB Imagescited by 234
  5. 2017BB8: A Scalable, Accurate, Robust to Partial Occlusion Method for Predicting the 3D Poses of Challenging Objects without Using Depthcited by 225
  6. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 225
  7. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  8. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
  9. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 214
  10. 2019PVN3D: A Deep Point-wise 3D Keypoints Voting Network for 6DoF Pose Estimationcited by 204
  11. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitationcited by 202
  12. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 194
  13. 20193D Hand Shape and Pose from Images in the Wildcited by 190
  14. 2021Where2Act: From Pixels to Actions for Articulated 3D Objectscited by 179
  15. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 175
  16. 20193D Hand Shape and Pose Estimation from a Single RGB Imagecited by 174
  17. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  18. 2022Human-to-Robot Imitation in the Wildcited by 157
  19. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  20. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  21. 2020Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wildcited by 146
  22. 2021Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Timecited by 140
  23. 2022EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relationscited by 139
  24. 2018Grounded Human-Object Interaction Hotspots from Videocited by 128
  25. 2019Pushing the Envelope for RGB-based Dense 3D Hand Pose Estimation via Neural Renderingcited by 115
  26. 2018Hand Pose Estimation via Latent 2.5D Heatmap Regressioncited by 111
  27. 2020Visual Imitation Made Easycited by 111
  28. 2022Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videoscited by 110
  29. 2023Affordance Diffusion: Synthesizing Hand-Object Interactionscited by 110
  30. 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105
  31. 2018Segmentation-driven 6D Object Pose Estimationcited by 104
  32. 2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
  33. 2023GenAug: Retargeting behaviors to unseen situations via Generative Augmentationcited by 103
  34. 2022VideoDex: Learning Dexterity from Internet Videoscited by 101
  35. 2018Cross-modal Deep Variational Hand Pose Estimationcited by 89
  36. 2019ContactGrasp: Functional Multi-finger Grasp Synthesis from Contactcited by 87
  37. 2021Learning by Watching: Physical Imitation of Manipulation Skills from Human Videoscited by 87
  38. 2022Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applicationscited by 78

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 20; 2025: 22; 2026: 9.

51 papers.

Sort:

By area:

  1. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
  2. 2024EgoMimic: Scaling Imitation Learning via Egocentric Videocited by 160
  3. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulationcited by 149
  4. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
  5. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 142
  6. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 116
  7. 2024Dreamitate: Real-World Visuomotor Policy Learning via Video Generationcited by 84
  8. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 80
  9. 2025Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learningcited by 57
  10. 2024RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulationcited by 55
  11. 2024Vision-based Manipulation from Single Human Video with Open-World Object Graphscited by 46
  12. 2025VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulationcited by 46
38 more
  1. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  2. 2024HRP: Human Affordances for Robotic Pre-Trainingcited by 40
  3. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 39
  4. 2024Hand-Object Interaction Pretraining from Videoscited by 38
  5. 2025NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videoscited by 38
  6. 2025CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learningcited by 37
  7. 2024ACE: A Cross-Platform Visual-Exoskeletons System for Low-Cost Dexterous Teleoperationcited by 32
  8. 2025ZeroMimic: Distilling Robotic Manipulation Skills from Web Videoscited by 31
  9. 2024DITTO: Demonstration Imitation by Trajectory Transformationcited by 28
  10. 2025X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Realcited by 25
  11. 2025TASTE-Rob: Advancing Video Generation of Task-Oriented Hand-Object Interaction for Generalizable Robotic Manipulationcited by 24
  12. 2024Mitigating the Human-Robot Domain Discrepancy in Visual Pre-training for Robotic Manipulationcited by 22
  13. 2024Diff-IP2D: Diffusion-Based Hand-Object Interaction Prediction on Egocentric Videoscited by 21
  14. 2025In-N-On: Scaling Egocentric Manipulation with in-the-wild and on-task Datacited by 21
  15. 2025Predicting 4D Hand Trajectory from Monocular Videoscited by 19
  16. 2024Semantically Controllable Augmentations for Generalizable Robot Learningcited by 17
  17. 2024HandsOnVLM: Vision-Language Models for Hand-Object Interaction Predictioncited by 15
  18. 2025Dexterity from Smart Lenses: Multi-Fingered Robot Manipulation with In-the-Wild Human Demonstrationscited by 14
  19. 2026MVP-LAM: Learning Action-Centric Latent Action via Cross-Viewpoint Reconstructioncited by 11
  20. 2024QuasiSim: Parameterized Quasi-Physical Simulators for Dexterous Manipulations Transfercited by 10
  21. 2026Dex4D: Task-Agnostic Point Track Policy for Sim-to-Real Dexterous Manipulationcited by 10
  22. 2025Human2LocoMan: Learning Versatile Quadrupedal Manipulation with Human Pretrainingcited by 8
  23. 2025H2R-Grounder: A Paired-Data-Free Paradigm for Translating Human Interaction Videos into Physically Grounded Robot Videoscited by 8
  24. 2025Chain-of-Modality: Learning Manipulation Programs from Multimodal Human Videos with Vision-Language-Modelscited by 6
  25. 2025OSMO: Open-Source Tactile Glove for Human-to-Robot Skill Transfercited by 6
  26. 2025Flowing from Reasoning to Motion: Learning 3D Hand Trajectory Prediction from Egocentric Human Interaction Videoscited by 6
  27. 2024Skills Made to Order: Efficient Acquisition of Robot Cooking Skills Guided by Multiple Forms of Internet Datacited by 4
  28. 2025Slot-Level Robotic Placement via Visual Imitation from Single Human Videocited by 2
  29. 2026What Matters When Cotraining Robot Manipulation Policies on Everyday Human Videos?cited by 2
  30. 2026Robot Self-Improvement via Human-Video Dynamics Modelscited by 2
  31. 2025GPA-RAM: Grasp-Pretraining Augmented Robotic Attention Mamba for Spatial Task Learningcited by 1
  32. 2025Leveraging Foundation Models for Multimodal Graph-Based Action Recognitioncited by 1
  33. 2025RoboSwap: A GAN-driven Video Diffusion Framework For Unsupervised Robot Arm Swappingcited by 1
  34. 2026Imitating What Works: Simulation-Filtered Modular Policy Learning from Human Videoscited by 1
  35. 2025Imitation Learning with Precisely Labeled Human Demonstrationscited by 0
  36. 2026Flow-Enabled Generalization to Human Demonstrations in Few-Shot Imitation Learningcited by 0
  37. 2026RoboPCA: Pose-centered Affordance Learning from Human Demonstrations for Robot Manipulationcited by 0
  38. 2026RAAP: Retrieval-Augmented Affordance Prediction with Cross-Image Action Alignmentcited by 0

50 most cited of 51 All 51 (JSON)