mlsanityData to 14 Oct 2026

Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation

arXiv 2403.04436 ↗First posted 7 Mar 2024cs.RO cs.AI cs.LGcited by 130

On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation

Abstract (arXiv)

We present Human to Humanoid (H2O), a reinforcement learning (RL) based framework that enables real-time whole-body teleoperation of a full-sized humanoid robot with only an RGB camera. To create a large-scale retargeted motion dataset of human movements for humanoid robots, we propose a scalable "sim-to-data" process to filter and pick feasible motions using a privileged motion imitator. Afterwards, we train a robust real-time humanoid motion imitator in simulation using these refined motions and transfer it to the real humanoid robot in a zero-shot manner. We successfully achieve teleoperation of dynamic whole-body motions in real-world scenarios, including walking, back jumping, kicking, turning, waving, pushing, boxing, etc. To the best of our knowledge, this is the first demonstration to achieve learning-based real-time whole-body humanoid teleoperation.

Key facts. Cited by 130 papers in the map (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 32 of its 61 references that are in the map. First posted on arXiv 7 Mar 2024.

Builds on

32 of 61 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Reinforcement Learning for Versatile, Dynamic, and Robust Bipedal Locomotion Control30 Jan

Other references · 31

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,568
  4. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,236
  5. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
  6. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 512
  7. 2024Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robotscited by 394
  8. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 391
  9. 2020HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimationcited by 307
  10. 2023Perpetual Humanoid Control for Real-time Simulated Avatarscited by 218
  11. 2018DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skillscited by 163
  12. 2024Expressive Whole-Body Control for Humanoid Robotscited by 139
19 more
  1. 2023Universal Humanoid Motion Representations for Physics-Based Controlcited by 116
  2. 2021Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimationcited by 101
  3. 2021Blind Bipedal Stair Traversal via Sim-to-Real Reinforcement Learningcited by 92
  4. 2022Touch and Go: Learning from Human-Collected Vision and Touchcited by 87
  5. 2022QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatarscited by 77
  6. 2024Agile But Safe: Learning Collision-Free High-Speed Legged Locomotioncited by 73
  7. 2024Humanoid Locomotion as Next Token Predictioncited by 69
  8. 2020Residual Force Control for Agile Human Behavior Imitation and Extended Motion Synthesiscited by 64
  9. 2023Deep Imitation Learning for Humanoid Loco-manipulation through Human Teleoperationcited by 54
  10. 2020UniCon: Universal Neural Controller For Physics-based Character Motioncited by 40
  11. 2023Teleoperation of Humanoid Robots: A Surveycited by 34
  12. 2023HumanMimic: Learning Natural Locomotion and Transitions for Humanoid Robot via Wasserstein Adversarial Imitationcited by 32
  13. 2022Imitate and Repurpose: Learning Reusable Robot Movement Skills From Human and Animal Behaviorscited by 30
  14. 2019Whole-Body Geometric Retargeting for Humanoid Robotscited by 20
  15. 2022Learning and Deploying Robust Locomotion Policies with Minimal Dynamics Randomizationcited by 17
  16. 2022iCub3 Avatar System: Enabling Remote Fully-Immersive Embodiment of Humanoid Robotscited by 15
  17. 2023SLoMo: A General System for Legged Robot Motion Imitation from Casual Videoscited by 12
  18. 2018Telexistence and Teleoperation for Walking Humanoid Robotscited by 5
  19. 2023NimbRo wins ANA Avatar XPRIZE Immersive Telepresence Competition: Human-Centric Evaluation and Lessons Learnedcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 26; 2025: 59; 2026: 45.

130 papers.

Sort:

By area:

  1. 2024OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learningcited by 189
  2. 2024HumanPlus: Humanoid Shadowing and Imitation from Humanscited by 185
  3. 2024Open-TeleVision: Teleoperation with Immersive Active Visual Feedbackcited by 145
  4. 2025ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skillscited by 130
  5. 2024DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learningcited by 116
  6. 2025TWIST: Teleoperated Whole-Body Imitation Systemcited by 101
  7. 2024ExBody2: Advanced Expressive Humanoid Whole-Body Controlcited by 88
  8. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  9. 2025HOMIE: Humanoid Loco-Manipulation with Isomorphic Exoskeleton Cockpitcited by 76
  10. 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 72
  11. 2025Visual Imitation Enables Contextual Humanoid Controlcited by 68
  12. 2024OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitationcited by 67
38 more
  1. 2025ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learningcited by 66
  2. 2024TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correctioncited by 59
  3. 2024Humanoid-Gym: Reinforcement Learning for Humanoid Robot with Zero-Shot Sim2Real Transfercited by 53
  4. 2024Generalizable Humanoid Manipulation with 3D Diffusion Policiescited by 52
  5. 2025KungfuBot: Physics-Based Humanoid Whole-Body Control for Learning Highly-Dynamic Skillscited by 52
  6. 2024Mobile-TeleVision: Predictive Motion Priors for Humanoid Whole-Body Controlcited by 51
  7. 2025AMO: Adaptive Motion Optimization for Hyper-Dexterous Humanoid Whole-Body Controlcited by 51
  8. 2025BeamDojo: Learning Agile Humanoid Locomotion on Sparse Footholdscited by 46
  9. 2025Humanoid-VLA: Towards Universal Humanoid Control with Visual Integrationcited by 45
  10. 2024CooHOI: Learning Cooperative Human-Object Interaction with Manipulated Object Dynamicscited by 44
  11. 2024CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangementcited by 44
  12. 2025Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrationscited by 38
  13. 2025WholeBodyVLA: Towards Unified Latent VLA for Whole-Body Loco-Manipulation Controlcited by 36
  14. 2024Model-Based Diffusion for Trajectory Optimizationcited by 32
  15. 2024Learning Smooth Humanoid Locomotion through Lipschitz-Constrained Policiescited by 30
  16. 2025Whole-Body Conditioned Egocentric Video Predictioncited by 30
  17. 2024Learning from Massive Human Videos for Universal Humanoid Pose Controlcited by 29
  18. 2025HuB: Learning Extreme Humanoid Balancecited by 28
  19. 2024Hierarchical World Models as Visual Whole-Body Humanoid Controllerscited by 27
  20. 2025Learning Getting-Up Policies for Real-World Humanoid Robotscited by 26
  21. 2025Humanoid Everyday: A Comprehensive Robotic Dataset for Open-World Humanoid Manipulationcited by 25
  22. 2024Harmon: Whole-Body Motion Generation of Humanoid Robots from Language Descriptionscited by 23
  23. 2024Local Policies Enable Zero-shot Long-horizon Manipulationcited by 22
  24. 2025Towards Immersive Human-X Interaction: A Real-Time Framework for Physically Plausible Motion Synthesiscited by 20
  25. 2024Real-Time Simulated Avatar from Head-Mounted Sensorscited by 17
  26. 2025From Experts to a Generalist: Toward General Whole-Body Control for Humanoid Robotscited by 17
  27. 2024Learning Multi-Modal Whole-Body Control for Real-World Humanoid Robotscited by 16
  28. 2024Mimicking-Bench: A Benchmark for Generalizable Humanoid-Scene Interaction Learning via Human Mimickingcited by 16
  29. 2025DreamControl: Human-Inspired Whole-Body Humanoid Control for Scene Interaction via Guided Diffusioncited by 15
  30. 2025SoftMimic: Learning Compliant Whole-body Control from Examplescited by 14
  31. 2026TextOp: Real-time Interactive Text-Driven Humanoid Robot Motion Generation and Controlcited by 14
  32. 2025Bridging the Sim-to-Real Gap for Athletic Loco-Manipulationcited by 13
  33. 2025Learning to Transfer Human Hand Skills for Robot Manipulationscited by 11
  34. 2025A Unified and General Humanoid Whole-Body Controller for Versatile Locomotioncited by 11
  35. 2025RL from Physical Feedback: Aligning Large Motion Models with Humanoid Controlcited by 11
  36. 2025Dexplore: Scalable Neural Control for Dexterous Manipulation from Reference-Scoped Explorationcited by 11
  37. 2026FRoM-W1: Towards General Humanoid Whole-Body Control with Language Instructionscited by 10
  38. 2024FIction: 4D Future Interaction Prediction from Videocited by 9

50 most cited of 130 All 130 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning Adaptive Neural Teleoperation for Humanoid Robots: From Inverse Kinematics to End-to-End Control ↗
  2. 2025HDMI: Learning Interactive Humanoid Whole-Body Control from Human Videos
  3. 2026ZeroWBC: Learning Natural Whole-Body Humanoid Interaction from Human Egocentric Data
  4. 2026Humanoid Manipulation Interface: Humanoid Whole-Body Manipulation from Robot-Free Demonstrations
  5. 2025TWIST2: Scalable, Portable, and Holistic Humanoid Data Collection System
  6. 2026CLOT: Closed-Loop Global Motion Tracking for Whole-Body Humanoid Teleoperation
  7. 2026Whole-Body UMI: Transferring UMI Manipulation Skills to Humanoid Whole-Body Manipulation via Real-Time Motion Generation ↗
  8. 2025Robot Trains Robot: Automatic Real-World Policy Adaptation and Learning for Humanoids