mlsanityData to 14 Oct 2026

OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning

arXiv 2406.08858 ↗First posted 13 Jun 2024cs.RO cs.CV cs.LGcited by 189

Not on the map. Why?

Cited by 40 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present OmniH2O (Omni Human-to-Humanoid), a learning-based system for whole-body humanoid teleoperation and autonomy. Using kinematic pose as a universal control interface, OmniH2O enables various ways for a human to control a full-sized humanoid with dexterous hands, including using real-time teleoperation through VR headset, verbal instruction, and RGB camera. OmniH2O also enables full autonomy by learning from teleoperated demonstrations or integrating with frontier models such as GPT-4. OmniH2O demonstrates versatility and dexterity in various real-world whole-body tasks through teleoperation or autonomy, such as playing multiple sports, moving and manipulating objects, and interacting with humans. We develop an RL-based sim-to-real pipeline, which involves large-scale retargeting and augmentation of human motion datasets, learning a real-world deployable policy with sparse sensor input by imitating a privileged teacher policy, and reward designs to enhance robustness and stability. We release the first humanoid whole-body control dataset, OmniH2O-6, containing six everyday tasks, and demonstrate humanoid whole-body skill learning from teleoperated datasets.

Key facts. Cited by 189 papers in the map (data through 14 Oct 2026). Builds on 40 of its 63 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

40 of 63 references.

References · 40

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  6. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  7. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  8. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  9. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  10. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,236
  11. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 862
  12. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 852
28 more
  1. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 657
  2. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 604
  3. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 512
  4. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 391
  5. 2023Perpetual Humanoid Control for Real-time Simulated Avatarscited by 218
  6. 2023Real-World Humanoid Locomotion with Reinforcement Learningcited by 187
  7. 2024Expressive Whole-Body Control for Humanoid Robotscited by 139
  8. 2024Learning Human-to-Humanoid Real-Time Whole-Body Teleoperationcited by 130
  9. 2022Deep Whole-Body Control: Learning a Unified Policy for Manipulation and Locomotioncited by 121
  10. 2021Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robotscited by 116
  11. 2023Universal Humanoid Motion Representations for Physics-Based Controlcited by 116
  12. 2019DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm Systemcited by 114
  13. 2024Reinforcement Learning for Versatile, Dynamic, and Robust Bipedal Locomotion Controlcited by 99
  14. 2020Sim-to-Real Learning of All Common Bipedal Gaits via Periodic Reward Compositioncited by 96
  15. 2023LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learningcited by 79
  16. 2022QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatarscited by 77
  17. 2024Learning Visuotactile Skills with Two Multifingered Handscited by 76
  18. 2024Agile But Safe: Learning Collision-Free High-Speed Legged Locomotioncited by 73
  19. 2024Humanoid Locomotion as Next Token Predictioncited by 69
  20. 2023Deep Imitation Learning for Humanoid Loco-manipulation through Human Teleoperationcited by 54
  21. 2023Learning Vision-Based Bipedal Locomotion for Challenging Terraincited by 42
  22. 2023Teleoperation of Humanoid Robots: A Surveycited by 34
  23. 2023QuestEnvSim: Environment-Aware Simulated Motion Tracking from Sparse Sensorscited by 28
  24. 2022Neural3Points: Learning to Generate Physically Realistic Full-body Motion for Virtual Reality Userscited by 21
  25. 2022iCub3 Avatar System: Enabling Remote Fully-Immersive Embodiment of Humanoid Robotscited by 15
  26. 2023CAJun: Continuous Adaptive Jumping using a Learned Centroidal Controllercited by 15
  27. 2022Sim-to-Real Learning for Bipedal Locomotion Under Unsensed Dynamic Loadscited by 10
  28. 2018Telexistence and Teleoperation for Walking Humanoid Robotscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 18; 2025: 87; 2026: 84.

189 papers.

Sort:

By area:

  1. 2024EgoMimic: Scaling Imitation Learning via Egocentric Videocited by 175
  2. 2025BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusioncited by 165
  3. 2024Open-TeleVision: Teleoperation with Immersive Active Visual Feedbackcited by 145
  4. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  5. 2025ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skillscited by 130
  6. 2024DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learningcited by 116
  7. 2025TWIST: Teleoperated Whole-Body Imitation Systemcited by 101
  8. 2025SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Controlcited by 101
  9. 2025Humanoid Policy ~ Human Policycited by 78
  10. 2025HOMIE: Humanoid Loco-Manipulation with Isomorphic Exoskeleton Cockpitcited by 76
  11. 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 72
  12. 2025OmniRetarget: Interaction-Preserving Data Generation for Humanoid Whole-Body Loco-Manipulation and Scene Interactioncited by 71
38 more
  1. 2025ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learningcited by 66
  2. 2024Generalizable Humanoid Manipulation with 3D Diffusion Policiescited by 52
  3. 2025KungfuBot: Physics-Based Humanoid Whole-Body Control for Learning Highly-Dynamic Skillscited by 52
  4. 2024Mobile-TeleVision: Predictive Motion Priors for Humanoid Whole-Body Controlcited by 51
  5. 2025TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenizationcited by 51
  6. 2025Learning Humanoid Standing-up Control across Diverse Posturescited by 46
  7. 2025BeamDojo: Learning Agile Humanoid Locomotion on Sparse Footholdscited by 46
  8. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  9. 2025Humanoid-VLA: Towards Universal Humanoid Control with Visual Integrationcited by 45
  10. 2024CooHOI: Learning Cooperative Human-Object Interaction with Manipulated Object Dynamicscited by 44
  11. 2024CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangementcited by 44
  12. 2025TWIST2: Scalable, Portable, and Holistic Humanoid Data Collection Systemcited by 44
  13. 2025LeVERB: Humanoid Whole-Body Control with Latent Vision-Language Instructioncited by 43
  14. 2025ResMimic: From General Motion Tracking to Humanoid Whole-body Loco-Manipulation via Residual Learningcited by 41
  15. 2025WholeBodyVLA: Towards Unified Latent VLA for Whole-Body Loco-Manipulation Controlcited by 36
  16. 2025SPIDER: Scalable Physics-Informed Dexterous Retargetingcited by 32
  17. 2024ARCap: Collecting High-quality Human Demonstrations for Robot Learning with Augmented Reality Feedbackcited by 31
  18. 2025BEHAVIOR Robot Suite: Streamlining Real-World Whole-Body Manipulation for Everyday Household Activitiescited by 31
  19. 2024SkillMimic: Learning Basketball Interaction Skills from Demonstrationscited by 30
  20. 2024Learning Smooth Humanoid Locomotion through Lipschitz-Constrained Policiescited by 30
  21. 2025Whole-Body Conditioned Egocentric Video Predictioncited by 30
  22. 2024Learning from Massive Human Videos for Universal Humanoid Pose Controlcited by 29
  23. 2025HuB: Learning Extreme Humanoid Balancecited by 28
  24. 2025Learning Getting-Up Policies for Real-World Humanoid Robotscited by 26
  25. 2025SkillMimic-V2: Learning Robust and Generalizable Interaction Skills from Sparse and Noisy Demonstrationscited by 25
  26. 2025Humanoid Everyday: A Comprehensive Robotic Dataset for Open-World Humanoid Manipulationcited by 25
  27. 2025Being-0: A Humanoid Robotic Agent with Vision-Language Models and Modular Skillscited by 22
  28. 2025Physics-Driven Data Generation for Contact-Rich Manipulation via Trajectory Optimizationcited by 21
  29. 2025Towards Immersive Human-X Interaction: A Real-Time Framework for Physically Plausible Motion Synthesiscited by 20
  30. 2025ActiveUMI: Robotic Manipulation with Active Perception from Robot-Free Human Demonstrationscited by 19
  31. 2025PhysHSI: Towards a Real-World Generalizable and Natural Humanoid-Scene Interaction Systemcited by 18
  32. 2025From Language to Locomotion: Retargeting-free Humanoid Control via Motion Latent Guidancecited by 18
  33. 2025From Experts to a Generalist: Toward General Whole-Body Control for Humanoid Robotscited by 17
  34. 2024Mimicking-Bench: A Benchmark for Generalizable Humanoid-Scene Interaction Learning via Human Mimickingcited by 16
  35. 2025A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AIcited by 16
  36. 2024Learning Diverse Bimanual Dexterous Manipulation Skills from Human Demonstrationscited by 15
  37. 2024DexHandDiff: Interaction-aware Diffusion Planning for Adaptive Dexterous Manipulationcited by 15
  38. 2025DreamControl: Human-Inspired Whole-Body Humanoid Control for Scene Interaction via Guided Diffusioncited by 15

50 most cited of 189 All 189 (JSON)