# Deep Imitation Learning for Humanoid Loco-manipulation through Human Teleoperation

arXiv: https://arxiv.org/abs/2309.01952 · HTML: https://mlsanity.com/paper/2309.01952/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Placed in Real-world robot offline RL, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 19 of its 45 references that are in the map.
- First posted on arXiv 5 Sep 2023.

## Placed in

- Offline RL and imitation learning from data / [Real-world robot offline RL](https://mlsanity.com/offlinerl/real-world-robot-offline-rl/)

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation](https://mlsanity.com/paper/1710.04615/) (2017)
- [Critic Regularized Regression](https://mlsanity.com/paper/2006.15134/) (2020)
- [Deep Spatial Autoencoders for Visuomotor Learning](https://mlsanity.com/paper/1509.06113/) (2015)
- [VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priors](https://mlsanity.com/paper/2210.11339/) (2022)
- [Learning and Retrieval from Prior Data for Skill-based Imitation Learning](https://mlsanity.com/paper/2210.11435/) (2022)
- [Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulation](https://mlsanity.com/paper/2112.05251/) (2021)
- [Deep Imitation Learning for Bimanual Robotic Manipulation](https://mlsanity.com/paper/2010.05134/) (2020)
- [ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation](https://mlsanity.com/paper/2008.07792/) (2020)
- [Teleoperation of Humanoid Robots: A Survey](https://mlsanity.com/paper/2301.04317/) (2023)
- [Learning Multi-Arm Manipulation Through Collaborative Teleoperation](https://mlsanity.com/paper/2012.06738/) (2020)
- [Dynamic Mobile Manipulation via Whole-Body Bilateral Teleoperation of a Wheeled Humanoid](https://mlsanity.com/paper/2307.01350/) (2023)
- [Deploying the NASA Valkyrie Humanoid for IED Response: An Initial Approach and Evaluation Summary](https://mlsanity.com/paper/1910.00761/) (2019)
- [A Fast, Autonomous, Bipedal Walking Behavior over Rapid Regions](https://mlsanity.com/paper/2207.08312/) (2022)
- [Finding Locomanipulation Plans Quickly in the Locomotion Constrained Manifold](https://mlsanity.com/paper/1909.08804/) (2019)
- [Control and Evaluation of a Humanoid Robot with Rolling Contact Knees](https://mlsanity.com/paper/2210.00961/) (2022)

## Built on by (most cited first, 30 of 54)

- [Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation](https://mlsanity.com/paper/2401.02117/) (2024)
- [OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning](https://mlsanity.com/paper/2406.08858/) (2024)
- [HumanPlus: Humanoid Shadowing and Imitation from Humans](https://mlsanity.com/paper/2406.10454/) (2024)
- [Open-TeleVision: Teleoperation with Immersive Active Visual Feedback](https://mlsanity.com/paper/2407.01512/) (2024)
- [Expressive Whole-Body Control for Humanoid Robots](https://mlsanity.com/paper/2402.16796/) (2024)
- [Flow as the Cross-Domain Manipulation Interface](https://mlsanity.com/paper/2407.15208/) (2024)
- [Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation](https://mlsanity.com/paper/2403.04436/) (2024)
- [3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations](https://mlsanity.com/paper/2403.03954/) (2024)
- [DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning](https://mlsanity.com/paper/2410.24185/) (2024)
- [ExBody2: Advanced Expressive Humanoid Whole-Body Control](https://mlsanity.com/paper/2412.13196/) (2024)
- [Learning Visuotactile Skills with Two Multifingered Hands](https://mlsanity.com/paper/2404.16823/) (2024)
- [OmniRetarget: Interaction-Preserving Data Generation for Humanoid Whole-Body Loco-Manipulation and Scene Interaction](https://mlsanity.com/paper/2509.26633/) (2025)
- [OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitation](https://mlsanity.com/paper/2410.11792/) (2024)
- [Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models](https://mlsanity.com/paper/2504.11054/) (2025)
- [Generalizable Humanoid Manipulation with 3D Diffusion Policies](https://mlsanity.com/paper/2410.10803/) (2024)
- [Mobile-TeleVision: Predictive Motion Priors for Humanoid Whole-Body Control](https://mlsanity.com/paper/2412.07773/) (2024)
- [AMO: Adaptive Motion Optimization for Hyper-Dexterous Humanoid Whole-Body Control](https://mlsanity.com/paper/2505.03738/) (2025)
- [Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/2503.24361/) (2025)
- [CooHOI: Learning Cooperative Human-Object Interaction with Manipulated Object Dynamics](https://mlsanity.com/paper/2406.14558/) (2024)
- [BEHAVIOR Robot Suite: Streamlining Real-World Whole-Body Manipulation for Everyday Household Activities](https://mlsanity.com/paper/2503.05652/) (2025)
- [TeleMoMa: A Modular and Versatile Teleoperation System for Mobile Manipulation](https://mlsanity.com/paper/2403.07869/) (2024)
- [Learning from Massive Human Videos for Universal Humanoid Pose Control](https://mlsanity.com/paper/2412.14172/) (2024)
- [Humanoid Everyday: A Comprehensive Robotic Dataset for Open-World Humanoid Manipulation](https://mlsanity.com/paper/2510.08807/) (2025)
- [Mamba Policy: Towards Efficient 3D Diffusion Policy with Hybrid Selective State Models](https://mlsanity.com/paper/2409.07163/) (2024)
- [Dexterous Manipulation through Imitation Learning: A Survey](https://mlsanity.com/paper/2504.03515/) (2025)
- [Strategy and Skill Learning for Physics-based Table Tennis Animation](https://mlsanity.com/paper/2407.16210/) (2024)
- [Mimicking-Bench: A Benchmark for Generalizable Humanoid-Scene Interaction Learning via Human Mimicking](https://mlsanity.com/paper/2412.17730/) (2024)
- [Tool-as-Interface: Learning Robot Policies from Observing Human Tool Use](https://mlsanity.com/paper/2504.04612/) (2025)
- [XRoboToolkit: A Cross-Platform Framework for Robot Teleoperation](https://mlsanity.com/paper/2508.00097/) (2025)
- [Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactions](https://mlsanity.com/paper/2508.04681/) (2025)

## Abstract (arXiv)

We tackle the problem of developing humanoid loco-manipulation skills with deep imitation learning. The difficulty of collecting task demonstrations and training policies for humanoids with a high degree of freedom presents substantial challenges. We introduce TRILL, a data-efficient framework for training humanoid loco-manipulation policies from human demonstrations. In this framework, we collect human demonstration data through an intuitive Virtual Reality (VR) interface. We employ the whole-body control formulation to transform task-space commands by human operators into the robot's joint-torque actuation while stabilizing its dynamics. By employing high-level action abstractions tailored for humanoid loco-manipulation, our method can efficiently learn complex sensorimotor skills. We demonstrate the effectiveness of TRILL in simulation and on a real-world robot for performing various loco-manipulation tasks. Videos and additional materials can be found on the project page: https://ut-austin-rpl.github.io/TRILL.