mlsanityData to 7 Oct 2026

HOI-Diff: Text-Driven Synthesis of 3D Human-Object Interactions using Diffusion Models

arXiv 2312.06553 ↗First posted 11 Dec 2023cs.CVcited by 89

On the map: Diffusion models → Video generation

Abstract (arXiv)

We address the problem of generating realistic 3D human-object interactions (HOIs) driven by textual prompts. To this end, we take a modular design and decompose the complex task into simpler sub-tasks. We first develop a dual-branch diffusion model (HOI-DM) to generate both human and object motions conditioned on the input text, and encourage coherent motions by a cross-attention communication module between the human and object motion generation branches. We also develop an affordance prediction diffusion model (APDM) to predict the contacting area between the human and object during the interactions driven by the textual prompt. The APDM is independent of the results by the HOI-DM and thus can correct potential errors by the latter. Moreover, it stochastically generates the contacting points to diversify the generated motions. Finally, we incorporate the estimated contacting points into the classifier-guidance to achieve accurate and close contact between humans and objects. To train and evaluate our approach, we annotate BEHAVE dataset with text descriptions. Experimental results on BEHAVE and OMOMO demonstrate that our approach produces realistic HOIs with various interactions and different types of objects.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 48 of its 76 references that are in the map. First posted on arXiv 11 Dec 2023.

Builds on

48 of 76 references.

Closest ancestors · 14

Same direction, newest first

  1. 2023Hierarchical Generation of Human-Object Interactions with Diffusion Probabilistic Models3 Oct
  2. 2023TransFusion: A Practical and Effective Transformer-based Diffusion Model for 3D Human Motion Prediction30 Jul
  3. 2023TEDi: Temporally-Entangled Diffusion for Long-Term Motion Synthesis27 Jul
  4. 2023NIFTY: Neural Object Interaction Fields for Guided Human Motion Synthesis14 Jul
  5. 2023Guided Motion Diffusion for Controllable Human Motion Synthesis21 May
  6. 2023Make-An-Animation: Large-Scale Text-conditional 3D Human Motion Generation16 May
  7. 2023InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactions12 Apr
  8. 2023ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model3 Apr
  9. 2023Human Motion Diffusion as a Generative Prior2 Mar
  10. 2023Can We Use Diffusion Probabilistic Models for 3D Motion Prediction?28 Feb
  11. 2022Executing your Commands via Motion Diffusion in Latent Space8 Dec
  12. 2022BeLFusion: Latent Diffusion for Behavior-Driven Human Motion Prediction25 Nov
2 more
  1. 2022Human Motion Diffusion Model29 Sep
  2. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model31 Aug

Other references · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  11. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  12. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 577
22 more
  1. 2016The KIT Motion-Language Datasetcited by 334
  2. 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 246
  3. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 237
  4. 2023OmniControl: Control Any Joint at Any Time for Human Motion Generationcited by 233
  5. 2022MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesiscited by 218
  6. 2023Object Motion Guided Human Motion Synthesiscited by 206
  7. 2021Stochastic Scene-Aware Motion Predictioncited by 203
  8. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 181
  9. 2022HUMANISE: Language-conditioned Human Motion Generation in 3D Scenescited by 156
  10. 2021GOAL: Generating 4D Whole-Body Motion for Hand-Object Graspingcited by 142
  11. 2023Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusioncited by 137
  12. 2023CG-HOI: Contact-Guided 3D Human-Object Interaction Generationcited by 137
  13. 2022COUCH: Towards Controllable Human-Chair Interactionscited by 134
  14. 2022IMos: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactionscited by 121
  15. 2023Synthesizing Diverse Human Motions in 3D Indoor Scenescited by 109
  16. 20213D AffordanceNet: A Benchmark for Visual Object Affordance Understandingcited by 100
  17. 2022Towards Diverse and Natural Scene-aware 3D Human Motion Synthesiscited by 91
  18. 2021SAGA: Stochastic Whole-Body Grasping with Contactcited by 89
  19. 2023HumanMAC: Masked Motion Completion for Human Motion Predictioncited by 81
  20. 2023PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interactioncited by 72
  21. 2021O2O-Afford: Annotation-Free Large-Scale Object-Object Affordance Learningcited by 46
  22. 2023Open-Vocabulary Affordance Detection in 3D Point Cloudscited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 31; 2025: 37; 2026: 21.

89 papers.

Sort:

By area:

  1. 2024CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character controlcited by 103
  2. 2024Generating Human Interaction Motions in Scenes with Text Controlcited by 80
  3. 2024Rethinking Diffusion for Text-Driven Human Motion Generation: Redundant Representations, Evaluation, and Masked Autoregressioncited by 78
  4. 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 67
  5. 2024Large Motion Model for Unified Multi-Modal Motion Generationcited by 65
  6. 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 63
  7. 2024Human-Object Interaction from Human-Level Instructionscited by 62
  8. 2024ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactionscited by 54
  9. 2024HOI-M3:Capture Multiple Humans and Objects Interaction within Contextual Environmentcited by 52
  10. 2024THOR: Text to Human-Object Interaction Diffusion via Relation Interventioncited by 46
  11. 2024CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangementcited by 40
  12. 2024SMooDi: Stylized Motion Diffusion Modelcited by 37
38 more
  1. 2024HIMO: A New Benchmark for Full-Body Human Interacting with Multiple Objectscited by 34
  2. 2025InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generationcited by 31
  3. 2024The Dawn of Video Generation: Preliminary Explorations with SORA-like Modelscited by 30
  4. 2025HUMOTO: A 4D Dataset of Mocap Human Object Interactionscited by 30
  5. 2024BimArt: A Unified Approach for the Synthesis of 3D Bimanual Interaction with Articulated Objectscited by 29
  6. 2025Absolute Coordinates Make Motion Generation Easycited by 27
  7. 2024ZeroHSI: Zero-Shot 4D Human-Scene Interaction by Video Generationcited by 25
  8. 2025A Survey on Human Interaction Motion Generationcited by 25
  9. 2024F-HOI: Toward Fine-grained Semantic-Aligned 3D Human-Object Interactionscited by 24
  10. 2025Guiding Human-Object Interactions with Rich Geometry and Relationscited by 24
  11. 2024ManiDext: Hand-Object Manipulation Synthesis via Continuous Correspondence Embeddings and Residual-Guided Diffusioncited by 21
  12. 2025ChainHOI: Joint-based Kinematic Chain Modeling for Human-Object Interaction Generationcited by 19
  13. 2025DreamActor-H1: High-Fidelity Human-Product Demonstration Video Generation via Motion-designed Diffusion Transformerscited by 19
  14. 2024InterFusion: Text-Driven Generation of 3D Human-Object Interactioncited by 18
  15. 2024Pay Attention and Move Better: Harnessing Attention for Interactive Motion Generation and Training-free Editingcited by 17
  16. 2024Text-controlled Motion Mamba: Text-Instructed Temporal Grounding of Human Motioncited by 16
  17. 2025HOIDiNi: Human-Object Interaction through Diffusion Noise Optimizationcited by 16
  18. 2024Contact-aware Human Motion Generation from Textual Descriptionscited by 15
  19. 2024A Unified Diffusion Framework for Scene-aware Human Motion Estimation from Sparse Signalscited by 15
  20. 2024Mimicking-Bench: A Benchmark for Generalizable Humanoid-Scene Interaction Learning via Human Mimickingcited by 14
  21. 2024SCENIC: Scene-aware Semantic Navigation with Instruction-guided Controlcited by 13
  22. 2025InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsingcited by 13
  23. 2025HunyuanVideo-HOMA: Generic Human-Object Interaction in Multimodal Driven Human Animationcited by 13
  24. 2024SyncDiff: Synchronized Motion Diffusion for Multi-Body Human-Object Interaction Synthesiscited by 12
  25. 2025SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarioscited by 11
  26. 2025Human-Object Interaction via Automatically Designed VLM-Guided Motion Policycited by 10
  27. 2025CoDA: Coordinated Diffusion Noise Optimization for Whole-Body Manipulation of Articulated Objectscited by 10
  28. 2025HOI-Dyn: Learning Interaction Dynamics for Human-Object Motion Diffusioncited by 10
  29. 2026InterPrior: Scaling Generative Control for Physics-Based Human-Object Interactionscited by 10
  30. 2024DreamHOI: Subject-Driven Generation of 3D Human-Object Interactions with Diffusion Priorscited by 9
  31. 2025ScoreHOI: Physically Plausible Reconstruction of Human-Object Interaction via Score-Guided Diffusioncited by 9
  32. 2025Pulp Motion: Framing-aware multimodal camera and human motion generationcited by 8
  33. 2024FORCE: Physics-aware Human-object Interactioncited by 7
  34. 2025StyleMotif: Multi-Modal Motion Stylization using Style-Content Cross Fusioncited by 7
  35. 2025GenHOI: Generalizing Text-driven 4D Human-Object Interaction Synthesis for Unseen Objectscited by 7
  36. 2025Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactionscited by 7
  37. 2026OmniShow: Unifying Multimodal Conditions for Human-Object Interaction Video Generationcited by 7
  38. 20253D Human Interaction Generation: A Surveycited by 6

50 most cited of 89 All 89 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Controllable Human-Object Interaction Synthesis
  2. 2024OOD-HOI: Text-Driven 3D Whole-Body Human-Object Interactions Generation Beyond Training Domains
  3. 2025Auto-Regressive Diffusion for Generating 3D Human-Object Interactions
  4. 2024Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction
  5. 2026OneHOI: Unifying Human-Object Interaction Generation and Editing
  6. 2024AvatarGO: Zero-shot 4D Human-Object Interaction Generation and Animation
  7. 2023InteractDiffusion: Interaction Control in Text-to-Image Diffusion Models
  8. 2024HOIDiffusion: Generating Realistic 3D Hand-Object Interaction Data