mlsanityData to 7 Oct 2026

Hoi3DGen: Generating High-Quality Human-Object-Interactions in 3D

arXiv 2603.12126 ↗First posted 12 Mar 2026cs.CV cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Modeling and generating 3D human-object interactions from text is crucial for applications in AR, XR, and gaming. Existing approaches often rely on score distillation from text-to-image models, but their results suffer from the Janus problem and do not follow text prompts faithfully due to the scarcity of high-quality interaction data. We introduce Hoi3DGen, a framework that generates high-quality textured meshes of human-object interaction that follow the input interaction descriptions precisely. We first curate realistic and high-quality interaction data leveraging multimodal large language models, and then create a full text-to-3D pipeline, which achieves orders-of-magnitude improvements in interaction fidelity. Our method surpasses baselines by 4-15x in text consistency and 3-7x in 3D model quality, exhibiting strong generalization to diverse categories and interaction types, while maintaining high-quality 3D generation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 63 of its 76 references that are in the map. First posted on arXiv 12 Mar 2026.

Builds on

63 of 76 references.

References · 63

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023Segment Anythingcited by 8,158
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  6. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  7. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  8. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  9. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  10. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  11. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  12. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 928
38 more
  1. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  2. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 767
  3. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 742
  4. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 684
  5. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  6. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assetscited by 444
  7. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 397
  8. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  9. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Priorcited by 341
  10. 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 331
  11. 2023RealFusion: 360deg Reconstruction of Any Object from a Single Imagecited by 305
  12. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 294
  13. 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 246
  14. 2023Object Motion Guided Human Motion Synthesiscited by 206
  15. 2023Scalable 3D Captioning with Pretrained Modelscited by 201
  16. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 181
  17. 2023DreamAvatar: Text-and-Shape Guided 3D Human Avatar Generation via Diffusion Modelscited by 154
  18. 2023CG-HOI: Contact-Guided 3D Human-Object Interaction Generationcited by 137
  19. 2020CPF: Learning a Contact Potential Field to Model the Hand-Object Interactioncited by 127
  20. 2023Controllable Human-Object Interaction Synthesiscited by 107
  21. 2023DreamHuman: Animatable 3D Avatars from Textcited by 102
  22. 2022CHORE: Contact, Human and Object REconstruction from a single RGB imagecited by 95
  23. 2023HOI-Diff: Text-Driven Synthesis of 3D Human-Object Interactions using Diffusion Modelscited by 89
  24. 2023HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Videocited by 76
  25. 2023ChatPose: Chatting about 3D Human Posecited by 71
  26. 2022NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactionscited by 67
  27. 2024CameraHMR: Aligning People with Perspectivecited by 64
  28. 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 63
  29. 2024Human-Object Interaction from Human-Level Instructionscited by 62
  30. 2023AvatarVerse: High-quality & Stable 3D Avatar Creation from Text and Posecited by 60
  31. 2024ComboVerse: Compositional 3D Assets Creation Using Spatially-Aware Diffusion Guidancecited by 59
  32. 2023Visibility Aware Human-Object Interaction Tracking from Single RGB Cameracited by 58
  33. 2022Human-Aware Object Placement for Visual Environment Reconstructioncited by 56
  34. 2024IDOL: Instant Photorealistic 3D Human Creation from a Single Imagecited by 55
  35. 2023Instant3D: Instant Text-to-3D Generationcited by 45
  36. 2024Human-3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion Modelscited by 44
  37. 2023Template Free Reconstruction of Human-object Interaction with Procedural Interaction Generationcited by 37
  38. 2023Disentangled Clothed Avatar Generation from Text Descriptionscited by 33

50 most cited of 63 All 63 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024OOD-HOI: Text-Driven 3D Whole-Body Human-Object Interactions Generation Beyond Training Domains
  2. 2026Multimodal Priors-Augmented Text-Driven 3D Human-Object Interaction Generation
  3. 2023GenZI: Zero-Shot 3D Human-Scene Interaction Generation
  4. 2024HIMO: A New Benchmark for Full-Body Human Interacting with Multiple Objects
  5. 20253DGen-Bench: Comprehensive Benchmark Suite for 3D Generative Models
  6. 2025Auto-Regressive Diffusion for Generating 3D Human-Object Interactions
  7. 2025InterMoE: Individual-Specific 3D Human Interaction Generation via Dynamic Temporal-Selective MoE
  8. 2026InfBaGel: Human-Object-Scene Interaction Generation with Dynamic Perception and Iterative Refinement