mlsanityData to 14 Oct 2026

GenZI: Zero-Shot 3D Human-Scene Interaction Generation

arXiv 2311.17737 ↗First posted 29 Nov 2023cs.CV cs.GRcited by 40

Not on the map. Why?

Abstract (arXiv)

Can we synthesize 3D humans interacting with scenes without learning from any 3D human-scene interaction data? We propose GenZI, the first zero-shot approach to generating 3D human-scene interactions. Key to GenZI is our distillation of interaction priors from large vision-language models (VLMs), which have learned a rich semantic space of 2D human-scene compositions. Given a natural language description and a coarse point location of the desired interaction in a 3D scene, we first leverage VLMs to imagine plausible 2D human interactions inpainted into multiple rendered views of the scene. We then formulate a robust iterative optimization to synthesize the pose and shape of a 3D human model in the scene, guided by consistency with the 2D interaction hypotheses. In contrast to existing learning-based approaches, GenZI circumvents the conventional need for captured 3D interaction data, and allows for flexible control of the 3D interaction synthesis with easy-to-use text prompts. Extensive experiments show that our zero-shot approach has high flexibility and generality, making it applicable to diverse scene types, including both indoor and outdoor environments.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Builds on 38 of its 54 references that are in the map. First posted on arXiv 29 Nov 2023.

Builds on

38 of 54 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  10. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
  11. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,214
  12. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 938
26 more
  1. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 702
  2. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 627
  3. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 375
  4. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 295
  5. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 246
  6. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  7. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 239
  8. 2022AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Timecited by 200
  9. 2021FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integrationcited by 169
  10. 2020Populating 3D Scenes by Learning Human-Scene Interactioncited by 164
  11. 2022HUMANISE: Language-conditioned Human Motion Generation in 3D Scenescited by 158
  12. 2020Monocular Expressive Body Regression through Body-Driven Attentioncited by 155
  13. 2023SMPLer-X: Scaling Up Expressive Human Pose and Shape Estimationcited by 155
  14. 2022PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Imagescited by 153
  15. 2023One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformercited by 130
  16. 2019Generating 3D People in Scenes without Peoplecited by 127
  17. 2023State of the Art on Diffusion Models for Visual Computingcited by 127
  18. 2015Capturing Hands in Action using Discriminative Salient Points and Physics Simulationcited by 115
  19. 2023Affordance Diffusion: Synthesizing Hand-Object Interactionscited by 110
  20. 2023Synthesizing Physical Character-Scene Interactionscited by 104
  21. 2019Efficient Learning on Point Clouds with Basis Point Setscited by 103
  22. 2022Towards Diverse and Natural Scene-aware 3D Human Motion Synthesiscited by 91
  23. 2022Compositional Human-Scene Interaction Synthesis with Semantic Controlcited by 83
  24. 2019Putting Humans in a Scene: Learning Affordance in 3D Indoor Environmentscited by 72
  25. 2020Monocular Real-time Full Body Capture with Inter-part Correlationscited by 53
  26. 2023Narrator: Towards Natural Control of Human-Scene Interaction Generation via Relationship Reasoningcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 14; 2025: 19; 2026: 7.

40 papers.

Sort:

By area:

  1. 2024CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character controlcited by 109
  2. 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 72
  3. 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 64
  4. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 43
  5. 2024Beyond the Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained 2D Diffusion Modelscited by 32
  6. 2024BimArt: A Unified Approach for the Synthesis of 3D Bimanual Interaction with Articulated Objectscited by 30
  7. 2024ZeroHSI: Zero-Shot 4D Human-Scene Interaction by Video Generationcited by 25
  8. 2025A Survey on Human Interaction Motion Generationcited by 25
  9. 20253D Scene Generation: A Surveycited by 24
  10. 2024InterTrack: Tracking Human Object Interaction without Object Templatescited by 23
  11. 2025SuperDec: 3D Scene Decomposition with Superquadric Primitivescited by 21
  12. 2025InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsingcited by 13
28 more
  1. 2024You Think, You ACT: The New Task of Arbitrary Text to Motion Generationcited by 8
  2. 2024FORCE: Physics-aware Human-object Interactioncited by 7
  3. 2025Target-Aware Video Diffusion Modelscited by 7
  4. 2025HOI-PAGE: Zero-Shot Human-Object Interaction Generation with Part Affordance Guidancecited by 6
  5. 2025GenHSI: Controllable Generation of Human-Scene Interaction Videoscited by 6
  6. 2024On the Element-Wise Representation and Reasoning in Zero-Shot Image Recognition: A Systematic Surveycited by 5
  7. 2025HOSIG: Full-Body Human-Object-Scene Interaction Generation with Hierarchical Scene Perceptioncited by 5
  8. 2025VHOI: Controllable Video Generation of Human-Object Interactions from Sparse Trajectories via Motion Densificationcited by 4
  9. 2024COLLAGE: Collaborative Human-Agent Interaction Generation using Hierarchical Latent Diffusion and Language Modelscited by 3
  10. 2025Animating the Uncaptured: Humanoid Mesh Animation with Video Diffusion Modelscited by 3
  11. 2025Decoupled Generative Modeling for Human-Object Interaction Synthesiscited by 3
  12. 2026Open-Vocabulary Functional 3D Human-Scene Interaction Generationcited by 3
  13. 2026InfBaGel: Human-Object-Scene Interaction Generation with Dynamic Perception and Iterative Refinementcited by 3
  14. 2024Automatic Scene Generation: State-of-the-Art Techniques, Models, Datasets, Challenges, and Future Prospectscited by 2
  15. 2025Jointly Understand Your Command and Intention:Reciprocal Co-Evolution between Scene-Aware 3D Human Motion Synthesis and Analysiscited by 2
  16. 2025Learning to Generate Human-Human-Object Interactions from Textual Descriptionscited by 2
  17. 2024Diverse 3D Human Pose Generation in Scenes based on Decoupled Structurecited by 1
  18. 20243DArticCyclists: Generating Synthetic Articulated 8D Pose-Controllable Cyclist Data for Computer Vision Applicationscited by 1
  19. 2025Populate-A-Scene: Affordance-Aware Human Video Generationcited by 1
  20. 2025Contact-Aware Amodal Completion for Human-Object Interaction via Multi-Regional Inpaintingcited by 1
  21. 2026InHabit: Leveraging Image Foundation Models for Scalable 3D Human Placementcited by 1
  22. 2026InterPhys: Physics-aware Human Motion Synthesis in a Dynamic Scenecited by 1
  23. 2025Towards Affordance-Aware Articulation Synthesis for Rigged Objectscited by 0
  24. 2025FantasyHSI: Video-Generation-Centric 4D Human Synthesis In Any Scene through A Graph-based Multi-Agent Frameworkcited by 0
  25. 2025SceneAdapt: Scene-aware Adaptation of Human Motion Diffusioncited by 0
  26. 2026ArtHOI: Articulated Human-Object Interaction Synthesis by 4D Reconstruction from Video Priorscited by 0
  27. 2026Before the Shutter: Aesthetic and Actionable Portrait Photography Planning in 3D Scenescited by 0
  28. 2026Efficient Human-Contact Representation for Human-Scene Interactioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ViHOI: Human-Object Interaction Synthesis with Visual Priors
  2. 2026Hoi3DGen: Generating High-Quality Human-Object-Interactions in 3D
  3. 2024DreamHOI: Subject-Driven Generation of 3D Human-Object Interactions with Diffusion Priors
  4. 2022Reconstructing Action-Conditioned Human-Object Interactions Using Commonsense Knowledge Priors
  5. 2024Ask, Pose, Unite: Scaling Data Acquisition for Close Interactions with Vision Language Models
  6. 2024Human-Aware 3D Scene Generation with Spatially-constrained Diffusion Models
  7. 2025InterPose: Learning to Generate Human-Object Interactions from Large-Scale Web Videos
  8. 2025ZING-3D: Zero-shot Incremental 3D Scene Graphs via Vision-Language Models