mlsanityData to 7 Oct 2026

THOR: Text to Human-Object Interaction Diffusion via Relation Intervention

arXiv 2403.11208 ↗First posted 17 Mar 2024cs.CVcited by 46

On the map: Diffusion models → Video generation

Abstract (arXiv)

This paper addresses new methodologies to deal with the challenging task of generating dynamic Human-Object Interactions from textual descriptions (Text2HOI). While most existing works assume interactions with limited body parts or static objects, our task involves addressing the variation in human motion, the diversity of object shapes, and the semantic vagueness of object motion simultaneously. To tackle this, we propose a novel Text-guided Human-Object Interaction diffusion model with Relation Intervention (THOR). THOR is a cohesive diffusion model equipped with a relation intervention mechanism. In each diffusion step, we initiate text-guided human and object motion and then leverage human-object relations to intervene in object motion. This intervention enhances the spatial-temporal relations between humans and objects, with human-centric interaction representation providing additional guidance for synthesizing consistent motion from text. To achieve more reasonable and realistic results, interaction losses is introduced at different levels of motion granularity. Moreover, we construct Text-BEHAVE, a Text2HOI dataset that seamlessly integrates textual descriptions with the currently largest publicly available 3D HOI dataset. Both quantitative and qualitative experiments demonstrate the effectiveness of our proposed model.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 71 of its 86 references that are in the map. First posted on arXiv 17 Mar 2024.

Builds on

71 of 86 references.

Closest ancestors · 14

Same direction, newest first

  1. 2023HOI-Diff: Text-Driven Synthesis of 3D Human-Object Interactions using Diffusion Models11 Dec
  2. 2023Controllable Human-Object Interaction Synthesis6 Dec
  3. 2023Hierarchical Generation of Human-Object Interactions with Diffusion Probabilistic Models3 Oct
  4. 2023Fg-T2M: Fine-Grained Text-Driven Human Motion Generation via Diffusion Model12 Sep
  5. 2023Synthesizing Long-Term Human Motions with Diffusion Models via Coherent Sampling3 Aug
  6. 2023NIFTY: Neural Object Interaction Fields for Guided Human Motion Synthesis14 Jul
  7. 2023InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactions12 Apr
  8. 2023ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model3 Apr
  9. 2023Human Motion Diffusion as a Generative Prior2 Mar
  10. 2022Executing your Commands via Motion Diffusion in Latent Space8 Dec
  11. 2022PhysDiff: Physics-Guided Human Motion Diffusion Model5 Dec
  12. 2022Diffusion Motion: Generate Text-Guided 3D Human Motion by Diffusion Model22 Oct
2 more
  1. 2022Human Motion Diffusion Model29 Sep
  2. 2022FLAME: Free-form Language-based Motion Synthesis & Editing1 Sep

Other references · 57

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  5. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  6. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 577
  7. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 497
  8. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 486
  9. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
  10. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 406
  11. 2022TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Textscited by 357
  12. 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 246
38 more
  1. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 237
  2. 2023OmniControl: Control Any Joint at Any Time for Human Motion Generationcited by 233
  3. 2022MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesiscited by 218
  4. 2023Object Motion Guided Human Motion Synthesiscited by 206
  5. 2020Long-term Human Motion Prediction with Scene Contextcited by 191
  6. 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Modelscited by 188
  7. 2023One Transformer Fits All Distributions in Multi-Modal Diffusion at Scalecited by 185
  8. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 181
  9. 2022TEACH: Temporal Action Composition for 3D Humanscited by 173
  10. 2020Populating 3D Scenes by Learning Human-Scene Interactioncited by 164
  11. 2022HUMANISE: Language-conditioned Human Motion Generation in 3D Scenescited by 156
  12. 2020Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wildcited by 152
  13. 2021GOAL: Generating 4D Whole-Body Motion for Hand-Object Graspingcited by 142
  14. 2023CG-HOI: Contact-Guided 3D Human-Object Interaction Generationcited by 137
  15. 2022COUCH: Towards Controllable Human-Chair Interactionscited by 134
  16. 2022IMos: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactionscited by 121
  17. 2023AttT2M: Text-Driven Human Motion Generation with Multi-Perspective Attention Mechanismcited by 116
  18. 2021DanceFormer: Music Conditioned 3D Dance Generation with Parametric Motion Transformercited by 111
  19. 2023Synthesizing Diverse Human Motions in 3D Indoor Scenescited by 109
  20. 2023Synthesizing Physical Character-Scene Interactionscited by 102
  21. 2022CHORE: Contact, Human and Object REconstruction from a single RGB imagecited by 95
  22. 2022Ego-Body Pose Estimation via Ego-Head Pose Estimationcited by 93
  23. 2023Unified Human-Scene Interaction via Prompted Chain-of-Contactscited by 91
  24. 2022InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interactioncited by 90
  25. 2021SAGA: Stochastic Whole-Body Grasping with Contactcited by 89
  26. 2022Compositional Human-Scene Interaction Synthesis with Semantic Controlcited by 82
  27. 2022PoseGPT: Quantization-based 3D Human Motion Generation and Forecastingcited by 78
  28. 2022ActFormer: A GAN-based Transformer towards General Action-Conditioned 3D Human Motion Generationcited by 77
  29. 2022Full-Body Articulated Human-Object Interactioncited by 76
  30. 2023TM2D: Bimodality Driven 3D Dance Generation via Music-Text Integrationcited by 76
  31. 2022TOCH: Spatio-Temporal Object-to-Hand Correspondence for Motion Refinementcited by 69
  32. 2022NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactionscited by 67
  33. 2023SINC: Spatial Composition of 3D Human Motions for Simultaneous Action Generationcited by 62
  34. 2022Implicit Neural Representations for Variable Length Human Motion Generationcited by 61
  35. 2023Visibility Aware Human-Object Interaction Tracking from Single RGB Cameracited by 58
  36. 2022MIME: Human-Aware 3D Scene Generationcited by 57
  37. 2018Accurate and Diverse Sampling of Sequences based on a "Best of Many" Sample Objectivecited by 56
  38. 2022Human-Aware Object Placement for Visual Environment Reconstructioncited by 56

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 21; 2026: 10.

46 papers.

Sort:

By area:

  1. 2024CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character controlcited by 103
  2. 2024Rethinking Diffusion for Text-Driven Human Motion Generation: Redundant Representations, Evaluation, and Masked Autoregressioncited by 78
  3. 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 67
  4. 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 63
  5. 2024Human-Object Interaction from Human-Level Instructionscited by 62
  6. 2024CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangementcited by 40
  7. 2024SMooDi: Stylized Motion Diffusion Modelcited by 37
  8. 2025InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generationcited by 31
  9. 2025Absolute Coordinates Make Motion Generation Easycited by 27
  10. 2025Guiding Human-Object Interactions with Rich Geometry and Relationscited by 24
  11. 2025Towards Immersive Human-X Interaction: A Real-Time Framework for Physically Plausible Motion Synthesiscited by 20
  12. 2025ChainHOI: Joint-based Kinematic Chain Modeling for Human-Object Interaction Generationcited by 19
34 more
  1. 2024InterFusion: Text-Driven Generation of 3D Human-Object Interactioncited by 18
  2. 2024Pay Attention and Move Better: Harnessing Attention for Interactive Motion Generation and Training-free Editingcited by 17
  3. 2025OpenHOI: Open-World Hand-Object Interaction Synthesis with Multimodal Large Language Modelcited by 17
  4. 2025SemGeoMo: Dynamic Contextual Human Motion Generation with Semantic and Geometric Guidancecited by 16
  5. 2024A Unified Diffusion Framework for Scene-aware Human Motion Estimation from Sparse Signalscited by 15
  6. 2024SCENIC: Scene-aware Semantic Navigation with Instruction-guided Controlcited by 13
  7. 2024SyncDiff: Synchronized Motion Diffusion for Multi-Body Human-Object Interaction Synthesiscited by 12
  8. 2025Auto-Regressive Diffusion for Generating 3D Human-Object Interactionscited by 12
  9. 2024InterDyn: Controllable Interactive Dynamics with Video Diffusion Modelscited by 11
  10. 2025Human-Object Interaction via Automatically Designed VLM-Guided Motion Policycited by 10
  11. 2025HOI-Dyn: Learning Interaction Dynamics for Human-Object Motion Diffusioncited by 10
  12. 2024DreamHOI: Subject-Driven Generation of 3D Human-Object Interactions with Diffusion Priorscited by 9
  13. 2025ARFlow: Human Action-Reaction Flow Matching with Physical Guidancecited by 7
  14. 2025StyleMotif: Multi-Modal Motion Stylization using Style-Content Cross Fusioncited by 7
  15. 2025GenHOI: Generalizing Text-driven 4D Human-Object Interaction Synthesis for Unseen Objectscited by 7
  16. 20253D Human Interaction Generation: A Surveycited by 6
  17. 2026Unleashing Guidance Without Classifiers for Human-Object Interaction Animationcited by 5
  18. 2025DAViD: Modeling Dynamic Affordance of 3D Objects Using Pre-trained Video Diffusion Modelscited by 4
  19. 2025HOI-PAGE: Zero-Shot Human-Object Interaction Generation with Part Affordance Guidancecited by 4
  20. 2025Uni-Inter: Unifying 3D Human Motion Synthesis Across Diverse Interaction Contextscited by 4
  21. 2025SimGenHOI: Physically Realistic Whole-Body Humanoid-Object Interaction via Generative Modeling and Reinforcement Learningcited by 3
  22. 2026HAIC: Humanoid Agile Object Interaction Control via Dynamics-Aware World Modelcited by 3
  23. 2024COLLAGE: Collaborative Human-Agent Interaction Generation using Hierarchical Latent Diffusion and Language Modelscited by 2
  24. 2024MulSMo: Multimodal Stylized Motion Generation by Bidirectional Control Flowcited by 2
  25. 2026Multimodal Priors-Augmented Text-Driven 3D Human-Object Interaction Generationcited by 2
  26. 2025UniHM: Universal Human Motion Generation with Object Interactions in Indoor Scenescited by 1
  27. 2025Occlusion-Aware Temporally Consistent Amodal Completion for 3D Human-Object Interaction Reconstructioncited by 1
  28. 2026Uni-HOI:A Unified framework for Learning the Joint distribution of Text and Human-Object Interactioncited by 1
  29. 2026GIRAF: Towards Generalizable Human Interactions with Articulated Objectscited by 1
  30. 2026MaMi-HOI: Harmonizing Global Kinematics and Local Geometry for Human-Object Interaction Generationcited by 0
  31. 2026Dynamic Full-body Motion Agent with Object Interaction via Blending Pre-trained Modular Controllerscited by 0
  32. 2026Policy-as-Data: Learning Generalizable HOI Diffusion Models from Simulated Physicscited by 0
  33. 2026Harnessing Coupled Stream Completion For Human-Object Interaction Modelingcited by 0
  34. 2026PAMI: Part Anchored Motion for Text to Human-Object Interaction Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025HOIDiNi: Human-Object Interaction through Diffusion Noise Optimization
  2. 2026OneHOI: Unifying Human-Object Interaction Generation and Editing
  3. 2025Efficient Explicit Joint-level Interaction Modeling with Mamba for Text-guided HOI Generation
  4. 2024OOD-HOI: Text-Driven 3D Whole-Body Human-Object Interactions Generation Beyond Training Domains
  5. 2024Generating Human Interaction Motions in Scenes with Text Control
  6. 2024F-HOI: Toward Fine-grained Semantic-Aligned 3D Human-Object Interactions
  7. 2025Learning to Generate Human-Human-Object Interactions from Textual Descriptions
  8. 2024Generating Human Motion in 3D Scenes from Text Descriptions