# Recovering Physically Plausible Human-Object Interactions from Monocular Videos

arXiv: https://arxiv.org/abs/2606.05359 · HTML: https://mlsanity.com/paper/2606.05359/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 47 of its 56 references that are in the map.
- First posted on arXiv 3 Jun 2026.

## Builds on (most cited first, 30 of 47)

- [GRAB: A Dataset of Whole-Body Human Grasping of Objects](https://mlsanity.com/paper/2008.11200/) (2020)
- [HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction](https://mlsanity.com/paper/2203.01577/) (2022)
- [BEHAVE: Dataset and Method for Tracking Human Object Interactions](https://mlsanity.com/paper/2204.06950/) (2022)
- [Perpetual Humanoid Control for Real-time Simulated Avatars](https://mlsanity.com/paper/2305.06456/) (2023)
- [Object Motion Guided Human Motion Synthesis](https://mlsanity.com/paper/2309.16237/) (2023)
- [BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusion](https://mlsanity.com/paper/2508.08241/) (2025)
- [Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild](https://mlsanity.com/paper/2007.15649/) (2020)
- [HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos](https://mlsanity.com/paper/2411.19167/) (2024)
- [MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpainting](https://mlsanity.com/paper/2409.14393/) (2024)
- [Synthesizing Physical Character-Scene Interactions](https://mlsanity.com/paper/2302.00883/) (2023)
- [SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Control](https://mlsanity.com/paper/2511.07820/) (2025)
- [CHORE: Contact, Human and Object REconstruction from a single RGB image](https://mlsanity.com/paper/2204.02445/) (2022)
- [InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interaction](https://mlsanity.com/paper/2209.12354/) (2022)
- [Full-Body Articulated Human-Object Interaction](https://mlsanity.com/paper/2212.10621/) (2022)
- [PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interaction](https://mlsanity.com/paper/2312.04393/) (2023)
- [InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactions](https://mlsanity.com/paper/2502.20390/) (2025)
- [NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactions](https://mlsanity.com/paper/2212.07626/) (2022)
- [ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning](https://mlsanity.com/paper/2503.21860/) (2025)
- [Visibility Aware Human-Object Interaction Tracking from Single RGB Camera](https://mlsanity.com/paper/2303.16479/) (2023)
- [ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactions](https://mlsanity.com/paper/2401.10232/) (2024)
- [Physically Plausible Full-Body Hand-Object Interaction Synthesis](https://mlsanity.com/paper/2309.07907/) (2023)
- [TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenization](https://mlsanity.com/paper/2503.19901/) (2025)
- [I'M HOI: Inertia-aware Monocular Capture of 3D Human-Object Interactions](https://mlsanity.com/paper/2312.08869/) (2023)
- [CooHOI: Learning Cooperative Human-Object Interaction with Manipulated Object Dynamics](https://mlsanity.com/paper/2406.14558/) (2024)
- [PMP: Learning to Physically Interact with Environments using Part-wise Motion Priors](https://mlsanity.com/paper/2305.03249/) (2023)
- [Template Free Reconstruction of Human-object Interaction with Procedural Interaction Generation](https://mlsanity.com/paper/2312.07063/) (2023)
- [PDP: Physics-Based Character Animation via Diffusion Policy](https://mlsanity.com/paper/2406.00960/) (2024)
- [HUMOTO: A 4D Dataset of Mocap Human Object Interactions](https://mlsanity.com/paper/2504.10414/) (2025)
- [SkillMimic: Learning Basketball Interaction Skills from Demonstrations](https://mlsanity.com/paper/2408.15270/) (2024)
- [Diffuse-CLoC: Guided Diffusion for Physics-based Character Look-ahead Control](https://mlsanity.com/paper/2503.11801/) (2025)

## Abstract (arXiv)

In this paper, we propose RePHO, a method to reconstruct physically plausible human-object interactions (HOI) from monocular videos. While existing kinematic-based approaches produce visually plausible motion, they often result in physically implausible artifacts such as interpenetration and object floating. To overcome these issues, we introduce a physics-guided reconstruction framework. We begin with a kinematic estimate and then refine it by training a policy with reinforcement learning (RL). This policy is optimized to reproduce the interaction in a physics simulator. Because kinematic estimates are typically noisy, naive RL training can fail. Therefore, we propose an adaptive sampling strategy with a dual self-updating mechanism that can identify the frames with the most informative and reliable kinematic reconstruction. Our process progressively improves reconstruction quality and yields physically consistent HOI sequences. We demonstrate our approach on two standard HOI benchmarks and achieve clear improvements in physical plausibility metrics over state-of-the-art methods. Project Page: https://dingbang777.github.io/RePHO/