# SUGAR: A Scalable Human-Video-Driven Generalizable Humanoid Loco-Manipulation Learning Framework

arXiv: https://arxiv.org/abs/2605.20373 · HTML: https://mlsanity.com/paper/2605.20373/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 54 references that are in the map.
- First posted on arXiv 19 May 2026.

## Builds on (most cited first, 30 of 49)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects](https://mlsanity.com/paper/2312.08344/) (2023)
- [SAM 3D: 3Dfy Anything in Images](https://mlsanity.com/paper/2511.16624/) (2025)
- [Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation](https://mlsanity.com/paper/2403.04436/) (2024)
- [MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpainting](https://mlsanity.com/paper/2409.14393/) (2024)
- [DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Grasping](https://mlsanity.com/paper/2502.20900/) (2025)
- [ExBody2: Advanced Expressive Humanoid Whole-Body Control](https://mlsanity.com/paper/2412.13196/) (2024)
- [HOMIE: Humanoid Loco-Manipulation with Isomorphic Exoskeleton Cockpit](https://mlsanity.com/paper/2502.13013/) (2025)
- [InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactions](https://mlsanity.com/paper/2502.20390/) (2025)
- [OmniRetarget: Interaction-Preserving Data Generation for Humanoid Whole-Body Loco-Manipulation and Scene Interaction](https://mlsanity.com/paper/2509.26633/) (2025)
- [SAM 3D Body: Robust Full-Body Human Mesh Recovery](https://mlsanity.com/paper/2602.15989/) (2026)
- [Visual Imitation Enables Contextual Humanoid Control](https://mlsanity.com/paper/2505.03729/) (2025)
- [OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitation](https://mlsanity.com/paper/2410.11792/) (2024)
- [Visual Whole-Body Control for Legged Loco-Manipulation](https://mlsanity.com/paper/2403.16967/) (2024)
- [KungfuBot: Physics-Based Humanoid Whole-Body Control for Learning Highly-Dynamic Skills](https://mlsanity.com/paper/2506.12851/) (2025)
- [AMO: Adaptive Motion Optimization for Hyper-Dexterous Humanoid Whole-Body Control](https://mlsanity.com/paper/2505.03738/) (2025)
- [Vision-based Manipulation from Single Human Video with Open-World Object Graphs](https://mlsanity.com/paper/2405.20321/) (2024)
- [TWIST2: Scalable, Portable, and Holistic Humanoid Data Collection System](https://mlsanity.com/paper/2511.02832/) (2025)
- [ResMimic: From General Motion Tracking to Humanoid Whole-body Loco-Manipulation via Residual Learning](https://mlsanity.com/paper/2510.05070/) (2025)
- [CLONE: Closed-Loop Whole-Body Humanoid Teleoperation for Long-Horizon Tasks](https://mlsanity.com/paper/2506.08931/) (2025)
- [HDMI: Learning Interactive Humanoid Whole-Body Control from Human Videos](https://mlsanity.com/paper/2509.16757/) (2025)
- [WholeBodyVLA: Towards Unified Latent VLA for Whole-Body Loco-Manipulation Control](https://mlsanity.com/paper/2512.11047/) (2025)
- [InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation](https://mlsanity.com/paper/2509.09555/) (2025)
- [SkillMimic: Learning Basketball Interaction Skills from Demonstrations](https://mlsanity.com/paper/2408.15270/) (2024)
- [Learning from Massive Human Videos for Universal Humanoid Pose Control](https://mlsanity.com/paper/2412.14172/) (2024)
- [Psi0: An Open Foundation Model Towards Universal Humanoid Loco-Manipulation](https://mlsanity.com/paper/2603.12263/) (2026)
- [VisualMimic: Visual Humanoid Loco-Manipulation via Motion Tracking and Generation](https://mlsanity.com/paper/2509.20322/) (2025)

## Built on by (most cited first, 3 of 3)

- [Human-Centric Intelligence in the Era of Foundation Models: A Survey](https://mlsanity.com/paper/2608.18184/) (2026)
- [GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitives](https://mlsanity.com/paper/2609.39601/) (2026)
- [From Video to Vision: Learning Generalizable Humanoid Carrying from Human Motion](https://mlsanity.com/paper/2610.14561/) (2026)

## Abstract (arXiv)

Building humanoid robots capable of generalizable whole-body loco-manipulation in the real world remains a fundamental challenge. Existing methods either rely on laborious task-specific reward engineering, rigidly replay reference motions that fail to generalize, or depend on costly teleoperation that limits scalability. While human videos capture diverse human behaviors, motion priors inferred from them are inherently imperfect, suffering from occlusion, contact artifacts, and retargeting errors that render them unsuitable for direct policy learning. To address this, we present SUGAR, a scalable data-driven framework that converts diverse human videos into deployable humanoid loco-manipulation skills, without any task-specific reward engineering or reference-motion conditioning at inference. SUGAR proceeds in three stages. First, a fully automated pipeline extracts kinematic interaction priors including human-object motion trajectories and contact labels from unstructured human videos. Second, a privileged physics-based refiner uses a unified mimic reward and progressive state pool to transform imperfect priors into physically feasible, high-fidelity skills. Third, refined skills are distilled into a hierarchical autonomous policy consisting of a command generator and a command tracker. We evaluate SUGAR on six representative loco-manipulation tasks in simulation and real-world humanoid hardware. Our method substantially outperforms reference-tracking baselines, and performance scales clearly with the amount of human video data. It also achieves zero-shot real-world transfer with reliable closed-loop execution, autonomous failure recovery, and stable long-horizon performance under external perturbations. Project Page: https://tianshuwu.github.io/sugar-humanoid/