# PhyGenHOI: Physically-Aware 4D Generation of Dynamic Human-Object Interactions

arXiv: https://arxiv.org/abs/2605.30268 · HTML: https://mlsanity.com/paper/2605.30268/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 32 references that are in the map.
- First posted on arXiv 28 May 2026.

## Builds on (most cited first, 28 of 28)

- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation](https://mlsanity.com/paper/2309.16653/) (2023)
- [Structured 3D Latents for Scalable and Versatile 3D Generation](https://mlsanity.com/paper/2412.01506/) (2024)
- [InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation](https://mlsanity.com/paper/2307.06942/) (2023)
- [Evaluating Text-to-Visual Generation with Image-to-Text Generation](https://mlsanity.com/paper/2404.01291/) (2024)
- [PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics](https://mlsanity.com/paper/2311.12198/) (2023)
- [GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models](https://mlsanity.com/paper/2310.08529/) (2023)
- [Object Motion Guided Human Motion Synthesis](https://mlsanity.com/paper/2309.16237/) (2023)
- [DreamGaussian4D: Generative 4D Gaussian Splatting](https://mlsanity.com/paper/2312.17142/) (2023)
- [4D-fy: Text-to-4D Generation Using Hybrid Score Distillation Sampling](https://mlsanity.com/paper/2311.17984/) (2023)
- [HUGS: Human Gaussian Splats](https://mlsanity.com/paper/2311.17910/) (2023)
- [Consistent4D: Consistent 360deg Dynamic Object Generation from Monocular Video](https://mlsanity.com/paper/2311.02848/) (2023)
- [Animate124: Animating One Image to 4D Dynamic Scene](https://mlsanity.com/paper/2311.14603/) (2023)
- [Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation](https://mlsanity.com/paper/2401.08559/) (2024)
- [InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction](https://mlsanity.com/paper/2403.19652/) (2024)
- [Animate3D: Animating Any 3D Model with Multi-view Video Diffusion](https://mlsanity.com/paper/2407.11398/) (2024)
- [DreamPhysics: Learning Physics-Based 3D Dynamics with Video Diffusion Priors](https://mlsanity.com/paper/2406.01476/) (2024)
- [Efficient Physics Simulation for 3D Scenes via MLLM-Guided Gaussian Splatting](https://mlsanity.com/paper/2411.12789/) (2024)
- [AnimateAnyMesh: A Feed-Forward 4D Foundation Model for Text-Driven Universal Mesh Animation](https://mlsanity.com/paper/2506.09982/) (2025)
- [AvatarGO: Zero-shot 4D Human-Object Interaction Generation and Animation](https://mlsanity.com/paper/2410.07164/) (2024)
- [HOIDiNi: Human-Object Interaction through Diffusion Noise Optimization](https://mlsanity.com/paper/2506.15625/) (2025)
- [SyncDiff: Synchronized Motion Diffusion for Multi-Body Human-Object Interaction Synthesis](https://mlsanity.com/paper/2412.20104/) (2024)
- [Articulated Kinematics Distillation from Video Diffusion Models](https://mlsanity.com/paper/2504.01204/) (2025)
- [Animus3D: Text-driven 3D Animation via Motion Score Distillation](https://mlsanity.com/paper/2512.12534/) (2025)
- [Choreographing a World of Dynamic Objects](https://mlsanity.com/paper/2601.04194/) (2026)

## Abstract (arXiv)

We address the task of generating physically accurate and visually faithful 4D Human-Object Interaction (HOI). Given a static 3D human and target object represented as 3D Gaussian Splats (3DGS), our goal is to synthesize dynamic scenes where the human actively engages with the object through actions, such as punching or kicking, in accordance with a given input text. To this end, we introduce PhyGenHOI, a novel framework that couples generative human motion with an explicit physical object simulation. We model the human as a semantic agent driven by a Motion Diffusion Model (MDM) and the object as a physical agent simulated via the Material Point Method (MPM), utilizing 3D Gaussians as a unified, differentiable representation. We supervise their interaction through three coupled mechanisms: (1) A Windowed Attraction Loss that temporally synchronizes generative motion to intercept the object; (2) A Contact-Driven Re-simulation step that triggers physically consistent momentum transfer upon impact; and (3) A Masked Video-SDS objective that injects video-based priors to enhance contact fidelity. Experiments show PhyGenHOI generates physically consistent 4D HOI across diverse actions, humans, and objects, outperforming baselines. Project page and videos: https://omerbenishu.github.io/PhyGenHOI/