# GenHSI: Controllable Generation of Human-Scene Interaction Videos arXiv: https://arxiv.org/abs/2506.19840 · HTML: https://mlsanity.com/paper/2506.19840/ ## Key facts - Cited by 6 papers in the map (data through 14 Oct 2026). - Placed in Video generation, one of 14 directions in Diffusion models. - Builds on 98 of its 116 references that are in the map. - First posted on arXiv 24 Jun 2025. ## Placed in - Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/) ## Builds on (most cited first, 30 of 98) - [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020) - [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023) - [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023) - [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023) - [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023) - [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024) - [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023) - [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022) - [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023) - [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022) - [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023) - [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019) - [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023) - [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023) - [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016) - [PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis](https://mlsanity.com/paper/2310.00426/) (2023) - [Structured 3D Latents for Scalable and Versatile 3D Generation](https://mlsanity.com/paper/2412.01506/) (2024) - [LTX-Video: Realtime Video Latent Diffusion](https://mlsanity.com/paper/2501.00103/) (2024) - [MotionCtrl: A Unified and Flexible Motion Controller for Video Generation](https://mlsanity.com/paper/2312.03641/) (2023) - [VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models](https://mlsanity.com/paper/2401.09047/) (2024) - [DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors](https://mlsanity.com/paper/2310.12190/) (2023) - [CameraCtrl: Enabling Camera Control for Text-to-Video Generation](https://mlsanity.com/paper/2404.02101/) (2024) - [LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models](https://mlsanity.com/paper/2309.15103/) (2023) - [HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation](https://mlsanity.com/paper/2011.14672/) (2020) - [MoGe: Unlocking Accurate Monocular Geometry Estimation for Open-Domain Images with Optimal Training Supervision](https://mlsanity.com/paper/2410.19115/) (2024) - [Open-Sora Plan: Open-Source Large Video Generation Model](https://mlsanity.com/paper/2412.00131/) (2024) - [Resolving 3D Human Pose Ambiguities with 3D Scene Constraints](https://mlsanity.com/paper/1908.06963/) (2019) - [DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing](https://mlsanity.com/paper/2306.14435/) (2023) ## Built on by (most cited first, 6 of 6) - [CoMoVi: Co-Generation of 3D Human Motions and Realistic Videos](https://mlsanity.com/paper/2601.10632/) (2026) - [UMO: Unified In-Context Learning Unlocks Motion Foundation Model Priors](https://mlsanity.com/paper/2603.15975/) (2026) - [VHOI: Controllable Video Generation of Human-Object Interactions from Sparse Trajectories via Motion Densification](https://mlsanity.com/paper/2512.09646/) (2025) - [Open-Vocabulary Functional 3D Human-Scene Interaction Generation](https://mlsanity.com/paper/2601.20835/) (2026) - [Ms. Forcing: Efficient Streaming Video Generation with Multi-Scale Patchification and Attention](https://mlsanity.com/paper/2607.20940/) (2026) - [HumanCLAW: Can Vision-Language Models Act Through a Body?](https://mlsanity.com/paper/2607.27180/) (2026) ## Abstract (arXiv) Large-scale pre-trained video diffusion models have exhibited remarkable capabilities in diverse video generation. However, existing solutions face several challenges in generating long videos with rich human-scene interactions (HSI), including unrealistic dynamics and affordance, lack of subject identity preservation, and the need for expensive training. To this end, we propose GenHSI, a training-free method for controllable generation of long HSI videos with 3D awareness. Taking inspiration from movie animation, we subdivide the video synthesis into three stages: (1) script writing, (2) pre-visualization, and (3) animation. Given an image of a scene and a character with a user description, we use these three stages to generate long videos that preserve human identity and provide rich and plausible HSI. Script writing converts a complex text prompt involving a chain of HSI into simple atomic actions that are used in the pre-visualization stage to generate 3D keyframes. To synthesize plausible human interaction poses in 3D keyframes, we utilize pre-trained 2D inpainting diffusion models to generate plausible 2D human interactions based on view canonicalization, which eliminates the need for multi-view fitting in previous works. We then extend these interactions to 3D using robust iterative optimization, informed by contact cues and reasoning from VLMs. Prompted by these 3D keyframes, the pretrained video diffusion models can better generate consistent long videos with plausible dynamics and affordance in a 3D-aware manner. We are the first to synthesize a long video sequence with a chain of HSI actions without training based on the image references of the scene and character. Experiments demonstrate that our method can generate HSI videos that effectively preserve scene content and character identity with plausible human-scene interaction from a single image scene.