mlsanityData to 14 Oct 2026

Visual Imitation Enables Contextual Humanoid Control

arXiv 2505.03729 ↗First posted 6 May 2025cs.RO cs.CVcited by 68

Not on the map. Why?

Abstract (arXiv)

How can we teach humanoids to climb staircases and sit on chairs using the surrounding environment context? Arguably, the simplest way is to just show them-casually capture a human motion video and feed it to humanoids. We introduce VIDEOMIMIC, a real-to-sim-to-real pipeline that mines everyday videos, jointly reconstructs the humans and the environment, and produces whole-body control policies for humanoid robots that perform the corresponding skills. We demonstrate the results of our pipeline on real humanoid robots, showing robust, repeatable contextual control such as staircase ascents and descents, sitting and standing from chairs and benches, as well as other dynamic whole-body skills-all from a single policy, conditioned on the environment and global root commands. VIDEOMIMIC offers a scalable path towards teaching humanoids to operate in diverse real-world environments.

Key facts. Cited by 68 papers in the map (data through 14 Oct 2026). Builds on 52 of its 67 references that are in the map. First posted on arXiv 6 May 2025.

Builds on

52 of 67 references.

References · 52

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  3. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  4. 2017End-to-end Recovery of Human Shape and Posecited by 885
  5. 2020Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humanscited by 634
  6. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 620
  7. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 604
  8. 2025Continuous 3D Perception Model with Persistent Statecited by 548
  9. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 512
  10. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 508
  11. 2019Learning agile and dynamic motor skills for legged robotscited by 504
  12. 2022ViTPose: Simple Vision Transformer Baselines for Human Pose Estimationcited by 456
38 more
  1. 2024MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motioncited by 442
  2. 2021Robust Motion In-betweeningcited by 229
  3. 2024MegaSaM: Accurate, Fast, and Robust Structure and Motion from Casual Dynamic Videoscited by 228
  4. 2023WHAM: Reconstructing World-grounded Humans with Accurate 3D Motioncited by 206
  5. 2023Real-World Humanoid Locomotion with Reinforcement Learningcited by 187
  6. 2021FAST-LIO2: Fast Direct LiDAR-inertial Odometrycited by 183
  7. 2022Capturing and Inferring Dense Full-Body Human-Scene Contactcited by 180
  8. 2018DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skillscited by 163
  9. 2021GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Camerascited by 153
  10. 2023ANYmal Parkour: Learning Agile Navigation for Quadrupedal Robotscited by 152
  11. 2022Legged Locomotion in Challenging Terrains using Egocentric Visioncited by 151
  12. 2023Decoupling Human and Camera Motion from Videos in the Wildcited by 142
  13. 2024Learning Human-to-Humanoid Real-Time Whole-Body Teleoperationcited by 130
  14. 2023Universal Humanoid Motion Representations for Physics-Based Controlcited by 116
  15. 2024MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpaintingcited by 116
  16. 2021SimPoE: Simulated Character Control for 3D Human Pose Estimationcited by 114
  17. 2024TRAM: Global Trajectory and Motion of 3D Humans from in-the-wild Videoscited by 106
  18. 2023Synthesizing Physical Character-Scene Interactionscited by 104
  19. 2023Neural Kernel Surface Reconstructioncited by 104
  20. 20182D/3D Pose Estimation and Action Recognition using Multitask Deep Learningcited by 103
  21. 2021Learning Motion Priors for 4D Human Body Capture in 3D Scenescited by 98
  22. 2024Expressive Whole-Body 3D Gaussian Avatarcited by 89
  23. 2024ExBody2: Advanced Expressive Humanoid Whole-Body Controlcited by 88
  24. 2024Align3R: Aligned Monocular Depth Estimation for Dynamic Videoscited by 79
  25. 2024Humanoid Locomotion as Next Token Predictioncited by 69
  26. 2024Estimating Body and Hand Motion in an Ego-sensed Worldcited by 58
  27. 2024GeoCalib: Learning Single-image Calibration with Geometric Optimizationcited by 52
  28. 2025TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenizationcited by 51
  29. 2024Learning Humanoid Locomotion with Perceptive Internal Modelcited by 50
  30. 2023SLOPER4D: A Scene-Aware Dataset for Global 4D Human Pose Estimation in Urban Environmentscited by 47
  31. 2025BeamDojo: Learning Agile Humanoid Locomotion on Sparse Footholdscited by 46
  32. 2024DextrAH-G: Pixels-to-Action Dexterous Arm-Hand Grasping with Geometric Fabricscited by 42
  33. 2022D&D: Learning Human Dynamics from Dynamic Cameracited by 41
  34. 2024Hand-Object Interaction Pretraining from Videoscited by 39
  35. 2023PACE: Human and Camera Motion Estimation from in-the-wild Videoscited by 34
  36. 2023On the Benefits of 3D Pose and Tracking for Human Action Recognitioncited by 32
  37. 2024Reconstructing People, Places, and Camerascited by 31
  38. 2025Joint Optimization for 4D Human-Scene Reconstruction in the Wildcited by 29

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 18; 2026: 50.

68 papers.

Sort:

By area:

  1. 2025Flow Matching Policy Gradientscited by 99
  2. 2025OmniRetarget: Interaction-Preserving Data Generation for Humanoid Whole-Body Loco-Manipulation and Scene Interactioncited by 71
  3. 2025TWIST2: Scalable, Portable, and Holistic Humanoid Data Collection Systemcited by 44
  4. 2025ResMimic: From General Motion Tracking to Humanoid Whole-body Loco-Manipulation via Residual Learningcited by 41
  5. 2025Human3R: Everyone Everywhere All at Oncecited by 37
  6. 2026Interactive World Simulator for Robot Policy Training and Evaluationcited by 31
  7. 2025GaussGym: An open-source real-to-sim framework for learning locomotion from pixelscited by 22
  8. 2025Robot Learning from a Physical World Modelcited by 20
  9. 2025PhysHSI: Towards a Real-World Generalizable and Natural Humanoid-Scene Interaction Systemcited by 18
  10. 2025CBF-RL: Safety Filtering Reinforcement Learning in Training with Control Barrier Functionscited by 17
  11. 2026Flow Policy Gradients for Robot Controlcited by 16
  12. 2025DreamControl: Human-Inspired Whole-Body Humanoid Control for Scene Interaction via Guided Diffusioncited by 15
38 more
  1. 2026HumanX: Toward Agile and Generalizable Humanoid Interaction Skills from Human Videoscited by 14
  2. 2026TextOp: Real-time Interactive Text-Driven Humanoid Robot Motion Generation and Controlcited by 14
  3. 2026ZEST: Zero-shot Embodied Skill Transfer for Athletic Robot Controlcited by 11
  4. 2025OXE-AugE: A Large-Scale Robot Augmentation of OXE for Scaling Cross-Embodiment Policy Learningcited by 10
  5. 2025Humanoid Goalkeeper: Learning from Position Conditioned Task-Motion Constraintscited by 9
  6. 2025Towards Adaptable Humanoid Control via Adaptive Motion Trackingcited by 8
  7. 2026ZeroWBC: Learning Natural Whole-Body Humanoid Interaction from Human Egocentric Datacited by 8
  8. 2026DuoMo: Dual Motion Diffusion for World-Space Human Reconstructioncited by 7
  9. 2025From Generated Human Videos to Physically Plausible Robot Trajectoriescited by 6
  10. 2026MeshMimic: Geometry-Aware Humanoid Motion Learning through 3D Scene Reconstructioncited by 6
  11. 2026Robot Learning from Human Videos: A Surveycited by 5
  12. 2026Towards Motion Turing Test: Evaluating Human-Likeness in Humanoid Robotscited by 4
  13. 2026OnlineHMR: Video-based Online World-Grounded Human Mesh Recoverycited by 4
  14. 2025MaskedManipulator: Versatile Whole-Body Manipulationcited by 3
  15. 2025Contrastive Representation Learning for Robust Sim-to-Real Transfer of Adaptive Humanoid Locomotioncited by 3
  16. 2026Learning Human-Like Badminton Skills for Humanoid Robotscited by 3
  17. 2026SafeFlow: Real-Time Text-Driven Humanoid Whole-Body Control via Physics-Guided Rectified Flow and Selective Safety Gatingcited by 3
  18. 2026SUGAR: A Scalable Human-Video-Driven Generalizable Humanoid Loco-Manipulation Learning Frameworkcited by 3
  19. 2025A Scalable Whole-body Motion Transfer via Implicit Kinodynamic Motion Retargetingcited by 2
  20. 2026PhysMoDPO: Physically-Plausible Humanoid Motion with Preference Optimizationcited by 2
  21. 2026HSImul3R: Physics-in-the-Loop Reconstruction of Simulation-Ready Human-Scene Interactionscited by 2
  22. 2026Imagine2Real: Towards Zero-shot Humanoid-Object Interaction via Video Generative Priorscited by 2
  23. 2026Direct Dynamic Retargeting for Humanoid Imitation Learning from Videoscited by 2
  24. 2026PhiZero: A World Model Built Around Physical Languagecited by 2
  25. 2026PDF-HR: Pose Distance Fields for Humanoid Robotscited by 1
  26. 2026Generalizing from References using a Multi-Task Reference and Goal-Driven RL Frameworkcited by 1
  27. 2026EmbodMocap: In-the-Wild 4D Human-Scene Reconstruction for Embodied Agentscited by 1
  28. 2026DreamControl-v2: Simpler and Scalable Autonomous Humanoid Skills via Trainable Guided Diffusion Priorscited by 1
  29. 2026Surflo: Consistent 3D Surface Flow Model with Global Statecited by 1
  30. 2026Supervise What Survives: Geometry-Guided VLA Adaptation from Synthetic Robot Videoscited by 1
  31. 2026ADP: Adversarial Dynamics Priors for Physically Grounded Humanoid Locomotioncited by 1
  32. 2026TANGO: Humanoid Navigation in Cluttered Environments with a Whole-Body Vision-Language-Action Modelcited by 1
  33. 2025CRISP: Contact-Guided Real2Sim from Monocular Video with Planar Scene Primitivescited by 0
  34. 2026HARMONI: Aligning Human and Scene Priors for Multi-View 4D Reconstructioncited by 0
  35. 2026Learn Weightlessness: Imitate Non-Self-Stabilizing Motions on Humanoid Robotcited by 0
  36. 2026Terrain Consistent Reference-Guided RL for Humanoid Navigation Autonomycited by 0
  37. 2026HOIST: Humanoid Optimization with Imitation and Sample-efficient Tuning for Manipulating Suspended Loadscited by 0
  38. 2026TROPHIES: Temporal Reconstruction of Places, Humans, and Cameras from Multi-view Videoscited by 0

50 most cited of 68 All 68 (JSON)