mlsanityData to 7 Oct 2026

OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitation

arXiv 2410.11792 ↗First posted 15 Oct 2024cs.RO cs.AI cs.CVcited by 62

Not on the map. Why?

Abstract (arXiv)

We study the problem of teaching humanoid robots manipulation skills by imitating from single video demonstrations. We introduce OKAMI, a method that generates a manipulation plan from a single RGB-D video and derives a policy for execution. At the heart of our approach is object-aware retargeting, which enables the humanoid robot to mimic the human motions in an RGB-D video while adjusting to different object locations during deployment. OKAMI uses open-world vision models to identify task-relevant objects and retarget the body motions and hand poses separately. Our experiments show that OKAMI achieves strong generalizations across varying visual and spatial conditions, outperforming the state-of-the-art baseline on open-world imitation from observation. Furthermore, OKAMI rollout trajectories are leveraged to train closed-loop visuomotor policies, which achieve an average success rate of 79.2% without the need for labor-intensive teleoperation. More videos can be found on our website https://ut-austin-rpl.github.io/OKAMI/.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Builds on 44 of its 67 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

44 of 67 references.

References · 44

  1. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  2. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  3. 2023Vision Transformers Need Registerscited by 844
  4. 2018Open3D: A Modern Library for 3D Data Processingcited by 717
  5. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  6. 2023MotionGPT: Human Motion as a Foreign Languagecited by 521
  7. 2023CoTracker: It is Better to Track Togethercited by 516
  8. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  9. 2023Humans in 4D: Reconstructing and Tracking Humans with Transformerscited by 344
  10. 2023Reconstructing Hands in 3D with Transformerscited by 298
  11. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 214
  12. 2023Perpetual Humanoid Control for Real-time Simulated Avatarscited by 198
32 more
  1. 2023Putting the Object Back into Video Object Segmentationcited by 182
  2. 2022Embodied Hands: Modeling and Capturing Hands and Bodies Togethercited by 179
  3. 2024HumanPlus: Humanoid Shadowing and Imitation from Humanscited by 164
  4. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  5. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulationcited by 149
  6. 2023Decoupling Human and Camera Motion from Videos in the Wildcited by 140
  7. 2024Open-TeleVision: Teleoperation with Immersive Active Visual Feedbackcited by 137
  8. 2023AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation Systemcited by 136
  9. 2022VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priorscited by 135
  10. 2024Expressive Whole-Body Control for Humanoid Robotscited by 117
  11. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representationscited by 115
  12. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrationscited by 114
  13. 2024Learning Human-to-Humanoid Real-Time Whole-Body Teleoperationcited by 112
  14. 2019The 2019 DAVIS Challenge on VOS: Unsupervised Multi-Object Segmentationcited by 95
  15. 2021Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstrationcited by 92
  16. 2024Learning Visuotactile Skills with Two Multifingered Handscited by 71
  17. 2022ViTPose++: Vision Transformer for Generic Body Pose Estimationcited by 65
  18. 2023Learning Generalizable Manipulation Policies with Object-Centric 3D Representationscited by 60
  19. 2023Deep Imitation Learning for Humanoid Loco-manipulation through Human Teleoperationcited by 49
  20. 2024Vision-based Manipulation from Single Human Video with Open-World Object Graphscited by 46
  21. 2022Demonstrate Once, Imitate Immediately (DOME): Learning Visual Servoing for One-Shot Imitation Learningcited by 36
  22. 2024DITTO: Demonstration Imitation by Trajectory Transformationcited by 28
  23. 2021Learning Multi-Stage Tasks with One Demonstration via Self-Replaycited by 21
  24. 2019Whole-Body Geometric Retargeting for Humanoid Robotscited by 19
  25. 2024Berkeley Humanoid: A Research Platform for Learning-based Controlcited by 19
  26. 2021NimbRo Avatar: Interactive Immersive Telepresence with Force-Feedback Telemanipulationcited by 13
  27. 2019Human to Robot Whole-Body Motion Transfercited by 8
  28. 2020Dynamic Movement Primitive based Motion Retargeting for Dual-Arm Sign Language Motionscited by 6
  29. 2023Out of Sight, Still in Mind: Reasoning and Planning about Unobserved Objects with Video Tracking Enabled Memory Modelscited by 5
  30. 2023One-shot Visual Imitation via Attributed Waypoints and Demonstration Augmentationcited by 4
  31. 2023Development of a Whole-body Work Imitation Learning System by a Biped and Bi-armed Humanoidcited by 4
  32. 2023Learning Multi-Step Manipulation Tasks from A Single Human Demonstrationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 31; 2026: 25.

62 papers.

Sort:

By area:

  1. 2024ExBody2: Advanced Expressive Humanoid Whole-Body Controlcited by 75
  2. 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 67
  3. 2025HOMIE: Humanoid Loco-Manipulation with Isomorphic Exoskeleton Cockpitcited by 65
  4. 2025Humanoid Policy ~ Human Policycited by 65
  5. 2025ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learningcited by 56
  6. 2025Learning Humanoid Standing-up Control across Diverse Posturescited by 34
  7. 2025ResMimic: From General Motion Tracking to Humanoid Whole-body Loco-Manipulation via Residual Learningcited by 34
  8. 2025Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrationscited by 33
  9. 2025ZeroMimic: Distilling Robotic Manipulation Skills from Web Videoscited by 31
  10. 2025BEHAVIOR Robot Suite: Streamlining Real-World Whole-Body Manipulation for Everyday Household Activitiescited by 29
  11. 2024Learning from Massive Human Videos for Universal Humanoid Pose Controlcited by 27
  12. 2025Crossing the Human-Robot Embodiment Gap with Sim-to-Real RL using One Human Demonstrationcited by 27
38 more
  1. 2024Bridging the Human to Robot Dexterity Gap through Object-Oriented Rewardscited by 25
  2. 2025X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Realcited by 25
  3. 2025SPIDER: Scalable Physics-Informed Dexterous Retargetingcited by 25
  4. 2025Reconstructing Humans with a Biomechanically Accurate Skeletoncited by 22
  5. 2025Being-0: A Humanoid Robotic Agent with Vision-Language Models and Modular Skillscited by 21
  6. 2025In-N-On: Scaling Egocentric Manipulation with in-the-wild and on-task Datacited by 21
  7. 2025Physics-Driven Data Generation for Contact-Rich Manipulation via Trajectory Optimizationcited by 20
  8. 2025A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AIcited by 16
  9. 2025DemoDiffusion: One-Shot Human Imitation using pre-trained Diffusion Policycited by 16
  10. 2024Mimicking-Bench: A Benchmark for Generalizable Humanoid-Scene Interaction Learning via Human Mimickingcited by 14
  11. 2025MimicFunc: Imitating Tool Manipulation from a Single Human Video via Functional Correspondencecited by 11
  12. 2026SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulationcited by 10
  13. 2026HEX: Humanoid-Aligned Experts for Cross-Embodiment Whole-Body Manipulationcited by 10
  14. 2026ULTRA: Unified Multimodal Control for Autonomous Humanoid Whole-Body Loco-Manipulationcited by 9
  15. 2026Do as I Do: Dexterous Manipulation Data from Everyday Human Videoscited by 9
  16. 2025H2R-Grounder: A Paired-Data-Free Paradigm for Translating Human Interaction Videos into Physically Grounded Robot Videoscited by 8
  17. 2026ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretrainingcited by 8
  18. 2025SLAC: Safe and Efficient Real-Robot Reinforcement Learning via Unsupervised Simulation Pre-Trainingcited by 7
  19. 2025DexMan: Learning Bimanual Dexterous Manipulation from Human and Generated Videoscited by 7
  20. 2025MoMaGen: Generating Demonstrations under Soft and Hard Constraints for Multi-Step Bimanual Mobile Manipulationcited by 7
  21. 2025Spatial RoboGrasp: Generalized Robotic Grasping Control Policycited by 6
  22. 2025X-Diffusion: Training Diffusion Policies on Cross-Embodiment Human Demonstrationscited by 6
  23. 2025OSMO: Open-Source Tactile Glove for Human-to-Robot Skill Transfercited by 6
  24. 2025SafeMimic: Towards Safe and Autonomous Human-to-Robot Imitation for Mobile Manipulationcited by 5
  25. 2024ARMOR: Egocentric Perception for Humanoid Robot Collision Avoidance and Motion Planningcited by 4
  26. 2025Uni-Hand: Universal Hand Motion Forecasting in Egocentric Viewscited by 4
  27. 2026HandEdit: A Unified Benchmark for Egocentric Human-to-Robot Dexterous Hand Image Editingcited by 4
  28. 2026Scaling Single Human Demonstrations for Imitation Learning using Generative Foundational Modelscited by 3
  29. 2026Towards Motion Turing Test: Evaluating Human-Likeness in Humanoid Robotscited by 3
  30. 2024Spatially Visual Perception for End-to-End Robotic Learningcited by 2
  31. 2025Learning Predictive Visuomotor Coordinationcited by 2
  32. 2025SKEL-CF: Coarse-to-Fine Biomechanical Skeleton and Surface Mesh Recoverycited by 2
  33. 2026EgoExo-WM: Unlocking Exo Video for Ego World Modelscited by 2
  34. 2026SeeTraceAct: Visibility-Aware Latent Planning from Cross-Embodiment Demonstration Videoscited by 2
  35. 2025MASH: Cooperative-Heterogeneous Multi-Agent Reinforcement Learning for Single Humanoid Robot Locomotioncited by 1
  36. 2026PEAR: Pixel-aligned Expressive humAn mesh Recoverycited by 1
  37. 2026Imitating What Works: Simulation-Filtered Modular Policy Learning from Human Videoscited by 1
  38. 2026SUGAR: A Scalable Human-Video-Driven Generalizable Humanoid Loco-Manipulation Learning Frameworkcited by 1

50 most cited of 62 All 62 (JSON)