# InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactions

arXiv: https://arxiv.org/abs/2502.20390 · HTML: https://mlsanity.com/paper/2502.20390/

## Key facts

- Cited by 72 papers in the map (data through 14 Oct 2026).
- Builds on 151 of its 207 references that are in the map.
- First posted on arXiv 27 Feb 2025.

## Builds on (most cited first, 30 of 151)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [GRAB: A Dataset of Whole-Body Human Grasping of Objects](https://mlsanity.com/paper/2008.11200/) (2020)
- [ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation](https://mlsanity.com/paper/2204.13662/) (2022)
- [BEHAVE: Dataset and Method for Tracking Human Object Interactions](https://mlsanity.com/paper/2204.06950/) (2022)
- [OmniControl: Control Any Joint at Any Time for Human Motion Generation](https://mlsanity.com/paper/2310.08580/) (2023)
- [Perpetual Humanoid Control for Real-time Simulated Avatars](https://mlsanity.com/paper/2305.06456/) (2023)
- [Object Motion Guided Human Motion Synthesis](https://mlsanity.com/paper/2309.16237/) (2023)
- [Stochastic Scene-Aware Motion Prediction](https://mlsanity.com/paper/2108.08284/) (2021)
- [OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning](https://mlsanity.com/paper/2406.08858/) (2024)
- [HumanPlus: Humanoid Shadowing and Imitation from Humans](https://mlsanity.com/paper/2406.10454/) (2024)
- [InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusion](https://mlsanity.com/paper/2308.16905/) (2023)
- [EgoMimic: Scaling Imitation Learning via Egocentric Video](https://mlsanity.com/paper/2410.24221/) (2024)
- [MotionLCM: Real-time Controllable Motion Generation via Latent Consistency Model](https://mlsanity.com/paper/2404.19759/) (2024)
- [Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild](https://mlsanity.com/paper/2007.15649/) (2020)
- [Open-TeleVision: Teleoperation with Immersive Active Visual Feedback](https://mlsanity.com/paper/2407.01512/) (2024)
- [GOAL: Generating 4D Whole-Body Motion for Hand-Object Grasping](https://mlsanity.com/paper/2112.11454/) (2021)
- [CG-HOI: Contact-Guided 3D Human-Object Interaction Generation](https://mlsanity.com/paper/2311.16097/) (2023)
- [Scaling Up Dynamic Human-Scene Interaction Modeling](https://mlsanity.com/paper/2403.08629/) (2024)
- [Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenes](https://mlsanity.com/paper/2012.05522/) (2020)
- [COUCH: Towards Controllable Human-Chair Interactions](https://mlsanity.com/paper/2205.00541/) (2022)
- [Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning](https://mlsanity.com/paper/2304.13653/) (2023)

## Built on by (most cited first, 30 of 72)

- [OmniRetarget: Interaction-Preserving Data Generation for Humanoid Whole-Body Loco-Manipulation and Scene Interaction](https://mlsanity.com/paper/2509.26633/) (2025)
- [KungfuBot: Physics-Based Humanoid Whole-Body Control for Learning Highly-Dynamic Skills](https://mlsanity.com/paper/2506.12851/) (2025)
- [TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenization](https://mlsanity.com/paper/2503.19901/) (2025)
- [ResMimic: From General Motion Tracking to Humanoid Whole-body Loco-Manipulation via Residual Learning](https://mlsanity.com/paper/2510.05070/) (2025)
- [InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation](https://mlsanity.com/paper/2509.09555/) (2025)
- [Absolute Coordinates Make Motion Generation Easy](https://mlsanity.com/paper/2505.19377/) (2025)
- [DreamActor-H1: High-Fidelity Human-Product Demonstration Video Generation via Motion-designed Diffusion Transformers](https://mlsanity.com/paper/2506.10568/) (2025)
- [DreamControl: Human-Inspired Whole-Body Humanoid Control for Scene Interaction via Guided Diffusion](https://mlsanity.com/paper/2509.14353/) (2025)
- [HumanX: Toward Agile and Generalizable Humanoid Interaction Skills from Human Videos](https://mlsanity.com/paper/2602.02473/) (2026)
- [Human-Object Interaction via Automatically Designed VLM-Guided Motion Policy](https://mlsanity.com/paper/2503.18349/) (2025)
- [SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarios](https://mlsanity.com/paper/2506.02444/) (2025)
- [MOSPA: Human Motion Generation Driven by Spatial Audio](https://mlsanity.com/paper/2507.11949/) (2025)
- [Dexplore: Scalable Neural Control for Dexterous Manipulation from Reference-Scoped Exploration](https://mlsanity.com/paper/2509.09671/) (2025)
- [DemoHLM: From One Demonstration to Generalizable Humanoid Loco-Manipulation](https://mlsanity.com/paper/2510.11258/) (2025)
- [ZEST: Zero-shot Embodied Skill Transfer for Athletic Robot Control](https://mlsanity.com/paper/2602.00401/) (2026)
- [InterPrior: Scaling Generative Control for Physics-Based Human-Object Interactions](https://mlsanity.com/paper/2602.06035/) (2026)
- [HOI-Dyn: Learning Interaction Dynamics for Human-Object Motion Diffusion](https://mlsanity.com/paper/2507.01737/) (2025)
- [FrankenMotion: Part-level Human Motion Generation and Composition](https://mlsanity.com/paper/2601.10909/) (2026)
- [ULTRA: Unified Multimodal Control for Autonomous Humanoid Whole-Body Loco-Manipulation](https://mlsanity.com/paper/2603.03279/) (2026)
- [Target-Aware Video Diffusion Models](https://mlsanity.com/paper/2503.18950/) (2025)
- [GenHOI: Generalizing Text-driven 4D Human-Object Interaction Synthesis for Unseen Objects](https://mlsanity.com/paper/2506.15483/) (2025)
- [Learning Whole-Body Human-Humanoid Interaction from Human-Human Demonstrations](https://mlsanity.com/paper/2601.09518/) (2026)
- [Unleashing Guidance Without Classifiers for Human-Object Interaction Animation](https://mlsanity.com/paper/2603.25734/) (2026)
- [InterAgent: Physics-based Multi-agent Command Execution via Diffusion on Interaction Graphs](https://mlsanity.com/paper/2512.07410/) (2025)
- [TeamHOI: Learning a Unified Policy for Cooperative Human-Object Interactions with Any Team Size](https://mlsanity.com/paper/2603.07988/) (2026)
- [UMO: Unified In-Context Learning Unlocks Motion Foundation Model Priors](https://mlsanity.com/paper/2603.15975/) (2026)
- [Half-Physics: Enabling Kinematic 3D Human Model with Physical Interactions](https://mlsanity.com/paper/2507.23778/) (2025)
- [Advances in 4D Representation: Geometry, Motion, and Interaction](https://mlsanity.com/paper/2510.19255/) (2025)
- [HarmoHOI: Harmonizing Appearance and 3D Motion for Multi-view Hand-Object Interaction Synthesis](https://mlsanity.com/paper/2607.17097/) (2026)
- [MaskedManipulator: Versatile Whole-Body Manipulation](https://mlsanity.com/paper/2505.19086/) (2025)

## Abstract (arXiv)

Achieving realistic simulations of humans interacting with a wide range of objects has long been a fundamental goal. Extending physics-based motion imitation to complex human-object interactions (HOIs) is challenging due to intricate human-object coupling, variability in object geometries, and artifacts in motion capture data, such as inaccurate contacts and limited hand detail. We introduce InterMimic, a framework that enables a single policy to robustly learn from hours of imperfect MoCap data covering diverse full-body interactions with dynamic and varied objects. Our key insight is to employ a curriculum strategy -- perfect first, then scale up. We first train subject-specific teacher policies to mimic, retarget, and refine motion capture data. Next, we distill these teachers into a student policy, with the teachers acting as online experts providing direct supervision, as well as high-quality references. Notably, we incorporate RL fine-tuning on the student policy to surpass mere demonstration replication and achieve higher-quality solutions. Our experiments demonstrate that InterMimic produces realistic and diverse interactions across multiple HOI datasets. The learned policy generalizes in a zero-shot manner and seamlessly integrates with kinematic generators, elevating the framework from mere imitation to generative modeling of complex human-object interactions.