# GenPHRI: Agentic Generative Simulation for Physical Human-Robot Interaction

arXiv: https://arxiv.org/abs/2604.08664 · HTML: https://mlsanity.com/paper/2604.08664/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 28 of its 41 references that are in the map.
- First posted on arXiv 9 Apr 2026.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 28 of 28)

- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Point Transformer V3: Simpler, Faster, Stronger](https://mlsanity.com/paper/2312.10035/) (2023)
- [PhysDiff: Physics-Guided Human Motion Diffusion Model](https://mlsanity.com/paper/2212.02500/) (2022)
- [MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations](https://mlsanity.com/paper/2310.17596/) (2023)
- [Holodeck: Language Guided Generation of 3D Embodied AI Environments](https://mlsanity.com/paper/2312.09067/) (2023)
- [RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation](https://mlsanity.com/paper/2311.01455/) (2023)
- [DreamGen: Unlocking Generalization in Robot Learning through Video World Models](https://mlsanity.com/paper/2505.12705/) (2025)
- [HUMANISE: Language-conditioned Human Motion Generation in 3D Scenes](https://mlsanity.com/paper/2210.09729/) (2022)
- [Unified Human-Scene Interaction via Prompted Chain-of-Contacts](https://mlsanity.com/paper/2309.07918/) (2023)
- [DrEureka: Language Model Guided Sim-To-Real Transfer](https://mlsanity.com/paper/2406.01967/) (2024)
- [DemoGen: Synthetic Demonstration Generation for Data-Efficient Visuomotor Policy Learning](https://mlsanity.com/paper/2502.16932/) (2025)
- [GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMs](https://mlsanity.com/paper/2410.03645/) (2024)
- [SceneSmith: Agentic Generation of Simulation-Ready Indoor Scenes](https://mlsanity.com/paper/2602.09153/) (2026)
- [Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpainting](https://mlsanity.com/paper/2411.09823/) (2024)
- [SMPLitex: A Generative Model and Dataset for 3D Human Texture Estimation from Single Image](https://mlsanity.com/paper/2309.01855/) (2023)
- [Learning to Collaborate from Simulation for Robot-Assisted Dressing](https://mlsanity.com/paper/1909.06682/) (2019)
- [SimWorld: An Open-ended Realistic Simulator for Autonomous Agents in Physical and Social Worlds](https://mlsanity.com/paper/2512.01078/) (2025)
- [ArticuBot: Learning Universal Articulated Object Manipulation Policy via Large Scale Simulation](https://mlsanity.com/paper/2503.03045/) (2025)
- [Feel the Bite: Robot-Assisted Inside-Mouth Bite Transfer using Robust Mouth Perception and Physical Interaction-Aware Control](https://mlsanity.com/paper/2403.04067/) (2024)
- [Assistive VR Gym: Interactions with Real People to Improve Virtual Assistive Robots](https://mlsanity.com/paper/2007.04959/) (2020)
- [RABBIT: A Robot-Assisted Bed Bathing System with Multimodal Perception and Integrated Compliance](https://mlsanity.com/paper/2401.15159/) (2024)
- [GRACE: Generalizing Robot-Assisted Caregiving with User Functionality Embeddings](https://mlsanity.com/paper/2501.17855/) (2025)
- [Fast SAM 3D Body: Accelerating SAM 3D Body for Real-Time Full-Body Human Mesh Recovery](https://mlsanity.com/paper/2603.15603/) (2026)
- [Force-Modulated Visual Policy for Robot-Assisted Dressing with Arm Motions](https://mlsanity.com/paper/2509.12741/) (2025)
- [ReGen: Generative Robot Simulation via Inverse Design](https://mlsanity.com/paper/2511.04769/) (2025)
- [OpenRoboCare: A Multimodal Multi-Task Expert Demonstration Dataset for Robot Caregiving](https://mlsanity.com/paper/2511.13707/) (2025)
- [Manip4Care: Robotic Manipulation of Human Limbs for Solving Assistive Tasks](https://mlsanity.com/paper/2508.02649/) (2025)

## Abstract (arXiv)

Real-world physical human-robot interaction (pHRI) development is costly and slow, while simulation requires substantial task-specific scene and motion authoring. We present GenPHRI, a generative simulation framework for producing deployment-ready pHRI scenarios entirely from natural-language task descriptions using vision-language model (VLM) agents. Generated scenarios contain a room with furniture, a task-appropriate human pose and placement, and a robot executing the requested interaction. Generator and critic agents iterate on each component, and an orchestrator agent controls cross-stage decisions. With GenPHRI, we generate 50 different assistive scenarios in simulation with no manual intervention and deploy two tasks in a 12-participant user study. Vision policies trained on these scenarios achieve zero-shot task completion rates of ~80% in real-world trials against ~90% in simulation, while producing behaviors that participants rate as consistent with textual task prompts. Further, we demonstrate direct execution of generated motion as a complementary, training-free deployment pathway that enables rapid real-world inspection of generated trajectories at the cost of contact coverage. We release GenPHRI as a platform for generating, training, and deploying pHRI tasks from textual prompts. Additional information may be found on our project website: https://rchi-lab.github.io/gen_phri/