# PhysHSI: Towards a Real-World Generalizable and Natural Humanoid-Scene Interaction System

arXiv: https://arxiv.org/abs/2510.11072 · HTML: https://mlsanity.com/paper/2510.11072/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 62 of its 74 references that are in the map.
- First posted on arXiv 13 Oct 2025.

## Builds on (most cited first, 30 of 62)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://mlsanity.com/paper/2503.14734/) (2025)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [Which Training Methods for GANs do actually Converge?](https://mlsanity.com/paper/1801.04406/) (2018)
- [Asymmetric Actor Critic for Image-Based Robot Learning](https://mlsanity.com/paper/1710.06542/) (2017)
- [Stochastic Scene-Aware Motion Prediction](https://mlsanity.com/paper/2108.08284/) (2021)
- [OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning](https://mlsanity.com/paper/2406.08858/) (2024)
- [HumanPlus: Humanoid Shadowing and Imitation from Humans](https://mlsanity.com/paper/2406.10454/) (2024)
- [FAST-LIO2: Fast Direct LiDAR-inertial Odometry](https://mlsanity.com/paper/2107.06829/) (2021)
- [BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusion](https://mlsanity.com/paper/2508.08241/) (2025)
- [Expressive Whole-Body Control for Humanoid Robots](https://mlsanity.com/paper/2402.16796/) (2024)
- [World-Grounded Human Motion Recovery via Gravity-View Coordinates](https://mlsanity.com/paper/2409.06662/) (2024)
- [ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skills](https://mlsanity.com/paper/2502.01143/) (2025)
- [Universal Humanoid Motion Representations for Physics-Based Control](https://mlsanity.com/paper/2310.04582/) (2023)
- [Synthesizing Physical Character-Scene Interactions](https://mlsanity.com/paper/2302.00883/) (2023)
- [TWIST: Teleoperated Whole-Body Imitation System](https://mlsanity.com/paper/2505.02833/) (2025)
- [Humanoid Parkour Learning](https://mlsanity.com/paper/2406.10759/) (2024)
- [Unified Human-Scene Interaction via Prompted Chain-of-Contacts](https://mlsanity.com/paper/2309.07918/) (2023)
- [Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions](https://mlsanity.com/paper/2203.15103/) (2022)
- [ExBody2: Advanced Expressive Humanoid Whole-Body Control](https://mlsanity.com/paper/2412.13196/) (2024)
- [Humanoid Policy ~ Human Policy](https://mlsanity.com/paper/2503.13441/) (2025)
- [HOMIE: Humanoid Loco-Manipulation with Isomorphic Exoskeleton Cockpit](https://mlsanity.com/paper/2502.13013/) (2025)
- [Real-Time Style Modelling of Human Locomotion via Feature-Wise Transformations and Local Motion Phases](https://mlsanity.com/paper/2201.04439/) (2022)
- [Visual Imitation Enables Contextual Humanoid Control](https://mlsanity.com/paper/2505.03729/) (2025)
- [Visual Whole-Body Control for Legged Loco-Manipulation](https://mlsanity.com/paper/2403.16967/) (2024)
- [WoCoCo: Learning Whole-Body Humanoid Control with Sequential Contacts](https://mlsanity.com/paper/2406.06005/) (2024)

## Built on by (most cited first, 18 of 18)

- [Perceptive Humanoid Parkour: Chaining Dynamic Human Skills via Motion Matching](https://mlsanity.com/paper/2602.15827/) (2026)
- [HumanX: Toward Agile and Generalizable Humanoid Interaction Skills from Human Videos](https://mlsanity.com/paper/2602.02473/) (2026)
- [ULTRA: Unified Multimodal Control for Autonomous Humanoid Whole-Body Loco-Manipulation](https://mlsanity.com/paper/2603.03279/) (2026)
- [TeamHOI: Learning a Unified Policy for Cooperative Human-Object Interactions with Any Team Size](https://mlsanity.com/paper/2603.07988/) (2026)
- [Advances in 4D Representation: Geometry, Motion, and Interaction](https://mlsanity.com/paper/2510.19255/) (2025)
- [Learning Human-Like Badminton Skills for Humanoid Robots](https://mlsanity.com/paper/2602.08370/) (2026)
- [SUGAR: A Scalable Human-Video-Driven Generalizable Humanoid Loco-Manipulation Learning Framework](https://mlsanity.com/paper/2605.20373/) (2026)
- [SplitAdapter: Load-Aware Humanoid Loco-Manipulation via Factorized Adaptation](https://mlsanity.com/paper/2606.03297/) (2026)
- [Imagine2Real: Towards Zero-shot Humanoid-Object Interaction via Video Generative Priors](https://mlsanity.com/paper/2605.22272/) (2026)
- [Scene and Human in One World: Reconstruction in a Feedforward Pass](https://mlsanity.com/paper/2606.27720/) (2026)
- [Scaling Behavior Foundation Model for Humanoid Robots](https://mlsanity.com/paper/2607.15163/) (2026)
- [AdaptManip: Learning Adaptive Whole-Body Object Lifting and Delivery with Online Recurrent State Estimation](https://mlsanity.com/paper/2602.14363/) (2026)
- [ReactiveBFM: Reactive Closed-Loop Motion Planning Towards Universal Humanoid Whole-Body Control](https://mlsanity.com/paper/2606.30362/) (2026)
- [Weave: Learning Whole-Body Dexterous Loco-Manipulation from Human-Object Interactions](https://mlsanity.com/paper/2609.16683/) (2026)
- [LUCID: Latent-Skill Unified Control via Imagined Dynamics for Long-Horizon Humanoid Loco-Manipulation](https://mlsanity.com/paper/2608.07746/) (2026)
- [HIGenNTO: Scalable Humanoid Interaction Generation via Noise-Space Trajectory Optimization](https://mlsanity.com/paper/2609.22611/) (2026)
- [InterEvolve: Test-Time Evolution of Reward Programs for Humanoid Loco-Manipulation](https://mlsanity.com/paper/2610.02196/) (2026)
- [InterMimicGen: Scaling Humanoid Loco-Manipulation through Self-Evolving Motion Imitation](https://mlsanity.com/paper/2610.06850/) (2026)

## Abstract (arXiv)

Deploying humanoid robots to interact with real-world environments--such as carrying objects or sitting on chairs--requires generalizable, lifelike motions and robust scene perception. Although prior approaches have advanced each capability individually, combining them in a unified system is still an ongoing challenge. In this work, we present a physical-world humanoid-scene interaction system, PhysHSI, that enables humanoids to autonomously perform diverse interaction tasks while maintaining natural and lifelike behaviors. PhysHSI comprises a simulation training pipeline and a real-world deployment system. In simulation, we adopt adversarial motion prior-based policy learning to imitate natural humanoid-scene interaction data across diverse scenarios, achieving both generalization and lifelike behaviors. For real-world deployment, we introduce a coarse-to-fine object localization module that combines LiDAR and camera inputs to provide continuous and robust scene perception. We validate PhysHSI on four representative interactive tasks--box carrying, sitting, lying, and standing up--in both simulation and real-world settings, demonstrating consistently high success rates, strong generalization across diverse task goals, and natural motion patterns.