mlsanityData to 14 Oct 2026

InterFusion: Text-Driven Generation of 3D Human-Object Interaction

arXiv 2403.15612 ↗First posted 22 Mar 2024cs.CVcited by 18

Not on the map. Why?

Abstract (arXiv)

In this study, we tackle the complex task of generating 3D human-object interactions (HOI) from textual descriptions in a zero-shot text-to-3D manner. We identify and address two key challenges: the unsatisfactory outcomes of direct text-to-3D methods in HOI, largely due to the lack of paired text-interaction data, and the inherent difficulties in simultaneously generating multiple concepts with complex spatial relationships. To effectively address these issues, we present InterFusion, a two-stage framework specifically designed for HOI generation. InterFusion involves human pose estimations derived from text as geometric priors, which simplifies the text-to-3D conversion process and introduces additional constraints for accurate object generation. At the first stage, InterFusion extracts 3D human poses from a synthesized image dataset depicting a wide range of interactions, subsequently mapping these poses to interaction descriptions. The second stage of InterFusion capitalizes on the latest developments in text-to-3D generation, enabling the production of realistic and high-quality 3D HOI scenes. This is achieved through a local-global optimization process, where the generation of human body and object is optimized separately, and jointly refined with a global optimization of the entire scene, ensuring a seamless and contextually coherent integration. Our experimental results affirm that InterFusion significantly outperforms existing state-of-the-art methods in 3D HOI generation.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 54 of its 71 references that are in the map. First posted on arXiv 22 Mar 2024.

Builds on

54 of 71 references.

References · 54

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  5. 2013Auto-Encoding Variational Bayescited by 4,053
  6. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
  7. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  8. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  9. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
  10. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,290
  11. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,236
  12. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,219
38 more
  1. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 938
  2. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 785
  3. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 624
  4. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  5. 2021Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fieldscited by 575
  6. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  7. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  8. 2021Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesiscited by 460
  9. 2021In-Place Scene Labelling and Understanding with Implicit Scene Representationcited by 415
  10. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 405
  11. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 334
  12. 2020Neural Scene Graphs for Dynamic Scenescited by 298
  13. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 295
  14. 2022CLIP-Mesh: Generating textured meshes from text using pretrained image-text modelscited by 281
  15. 2021BABEL: Bodies, Action and Behavior with English Labelscited by 265
  16. 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 249
  17. 2021Learning Object-Compositional Neural Radiance Field for Editable Scene Renderingcited by 228
  18. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 228
  19. 2018Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddingscited by 169
  20. 2020BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Imagescited by 166
  21. 2020Populating 3D Scenes by Learning Human-Scene Interactioncited by 164
  22. 2023DreamAvatar: Text-and-Shape Guided 3D Human Avatar Generation via Diffusion Modelscited by 154
  23. 2023CG-HOI: Contact-Guided 3D Human-Object Interaction Generationcited by 138
  24. 2021Collaborative Regression of Expressive Bodies using Moderationcited by 137
  25. 2023TextMesh: Generation of Realistic 3D Meshes From Text Promptscited by 135
  26. 2019Generating 3D People in Scenes without Peoplecited by 127
  27. 2023Vox-E: Text-guided Voxel Editing of 3D Objectscited by 115
  28. 2023Compositional 3D Scene Generation using Locally Conditioned Diffusioncited by 101
  29. 2023HOI-Diff: Text-Driven Synthesis of 3D Human-Object Interactions using Diffusion Modelscited by 93
  30. 2020Object-Centric Neural Scene Renderingcited by 90
  31. 2023AvatarCraft: Transforming Text into Neural Human Avatars with Parameterized Shape and Pose Controlcited by 84
  32. 2022Compositional Human-Scene Interaction Synthesis with Semantic Controlcited by 83
  33. 2023Set-the-Scene: Global-Local Training for Generating Controllable NeRF Scenescited by 75
  34. 2022Object-Compositional Neural Implicit Surfacescited by 74
  35. 2022Towards Implicit Text-Guided 3D Shape Generationcited by 68
  36. 2022DisCoScene: Spatially Disentangled Generative Radiance Fields for Controllable 3D-aware Scene Synthesiscited by 61
  37. 2022COAP: Compositional Articulated Occupancy of Peoplecited by 59
  38. 2023CompoNeRF: Text-guided Multi-object Compositional NeRF with Editable 3D Scene Layoutcited by 57

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 10; 2026: 3.

18 papers.

Sort:

By area:

  1. 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 72
  2. 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 64
  3. 2024HOI-M3:Capture Multiple Humans and Objects Interaction within Contextual Environmentcited by 53
  4. 2024AvatarGO: Zero-shot 4D Human-Object Interaction Generation and Animationcited by 25
  5. 2025A Survey on Human Interaction Motion Generationcited by 25
  6. 2024TriDi: Trilateral Diffusion of 3D Humans, Objects, and Interactionscited by 20
  7. 2025InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsingcited by 13
  8. 2024DreamHOI: Subject-Driven Generation of 3D Human-Object Interactions with Diffusion Priorscited by 9
  9. 2025VolumetricSMPL: A Neural Volumetric Body Model for Efficient Interactions, Contacts, and Collisionscited by 9
  10. 2025ECHO: Ego-Centric modeling of Human-Object interactionscited by 5
  11. 2025OnlineHOI: Towards Online Human-Object Interaction Generation and Perceptioncited by 3
  12. 2026Uni-HOI:A Unified framework for Learning the Joint distribution of Text and Human-Object Interactioncited by 2
6 more
  1. 2026THOM: Generating Physically Plausible Hand-Object Meshes From Textcited by 1
  2. 2025IKMo: Image-Keyframed Motion Generation with Trajectory-Pose Conditioned Motion Diffusion Modelcited by 0
  3. 2025TextMesh4D: Zero-shot Text-to-4D Mesh Generationcited by 0
  4. 2025CEIDM: A Controlled Entity and Interaction Diffusion Model for Enhanced Text-to-Image Generationcited by 0
  5. 2025AnchorHOI: Zero-shot Generation of 4D Human-Object Interaction via Anchor-based Prior Distillationcited by 0
  6. 2026Hoi3DGen: Generating High-Quality Human-Object-Interactions in 3Dcited by 0