mlsanityData to 14 Oct 2026

Hierarchical Generation of Human-Object Interactions with Diffusion Probabilistic Models

arXiv 2310.02242 ↗First posted 3 Oct 2023cs.CV cs.GRcited by 43

On the map: Diffusion models → Video generation

Abstract (arXiv)

This paper presents a novel approach to generating the 3D motion of a human interacting with a target object, with a focus on solving the challenge of synthesizing long-range and diverse motions, which could not be fulfilled by existing auto-regressive models or path planning-based methods. We propose a hierarchical generation framework to solve this challenge. Specifically, our framework first generates a set of milestones and then synthesizes the motion along them. Therefore, the long-range motion generation could be reduced to synthesizing several short motion sequences guided by milestones. The experiments on the NSM, COUCH, and SAMP datasets show that our approach outperforms previous methods by a large margin in both quality and diversity. The source code is available on our project page https://zju3dv.github.io/hghoi.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 63 of its 87 references that are in the map. First posted on arXiv 3 Oct 2023.

Builds on

63 of 87 references.

Closest ancestors · 7

Same direction, newest first

  1. 2022Executing your Commands via Motion Diffusion in Latent Space8 Dec
  2. 2022Pretrained Diffusion Models for Unified Human Motion Synthesis6 Dec
  3. 2022BeLFusion: Latent Diffusion for Behavior-Driven Human Motion Prediction25 Nov
  4. 2022Human Motion Diffusion Model29 Sep
  5. 2022FLAME: Free-form Language-based Motion Synthesis & Editing1 Sep
  6. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model31 Aug
  7. 2022Video Diffusion Models7 Apr

Other references · 56

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  8. 2022Classifier-Free Diffusion Guidancecited by 5,847
  9. 2022Generative Adversarial Networkscited by 5,627
  10. 2017Neural Discrete Representation Learningcited by 4,712
  11. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  12. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
38 more
  1. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  2. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  3. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  4. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,043
  5. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 501
  6. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 492
  7. 2017On human motion prediction using recurrent neural networkscited by 338
  8. 2022Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Modelscited by 334
  9. 2022EDGE: Editable Dance Generation From Musiccited by 287
  10. 2019Learning Trajectory Dependencies for Human Motion Predictioncited by 246
  11. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 239
  12. 2015Structural-RNN: Deep Learning on Spatio-Temporal Graphscited by 234
  13. 2022MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesiscited by 218
  14. 2022AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generationcited by 208
  15. 2021Stochastic Scene-Aware Motion Predictioncited by 203
  16. 2020Long-term Human Motion Prediction with Scene Contextcited by 192
  17. 2020DLow: Diversifying Latent Flows for Diverse Human Motion Predictioncited by 185
  18. 2021Character Controllers Using Motion VAEscited by 184
  19. 2021Synthesis of Compositional Animations from Textual Descriptionscited by 178
  20. 2017HP-GAN: Probabilistic 3D human motion prediction via GANcited by 176
  21. 2020History Repeats Itself: Human Motion Prediction via Motion Attentioncited by 165
  22. 2020Populating 3D Scenes by Learning Human-Scene Interactioncited by 164
  23. 2021GOAL: Generating 4D Whole-Body Motion for Hand-Object Graspingcited by 144
  24. 2020Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenescited by 135
  25. 2022COUCH: Towards Controllable Human-Chair Interactionscited by 135
  26. 2020Dynamic Multiscale Graph Neural Networks for 3D Skeleton-Based Human Motion Predictioncited by 132
  27. 2019Generating 3D People in Scenes without Peoplecited by 127
  28. 2020We are More than Our Joints: Predicting how 3D Bodies Movecited by 115
  29. 2018QuaterNet: A Quaternion-based Recurrent Model for Human Motioncited by 109
  30. 2023Synthesizing Physical Character-Scene Interactionscited by 104
  31. 2021SAGA: Stochastic Whole-Body Grasping with Contactcited by 91
  32. 2022Towards Diverse and Natural Scene-aware 3D Human Motion Synthesiscited by 91
  33. 2018MT-VAE: Learning Motion Transformations to Generate Multimodal Human Dynamicscited by 86
  34. 2022GIMO: Gaze-Informed Human Motion Prediction in Contextcited by 81
  35. 2022PoseGPT: Quantization-based 3D Human Motion Generation and Forecastingcited by 78
  36. 2020Convolutional Autoencoders for Human Motion Infillingcited by 73
  37. 2021Scene-aware Generative Network for Human Motion Synthesiscited by 70
  38. 2022NeMF: Neural Motion Fields for Kinematic Animationcited by 68

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 20; 2025: 13; 2026: 4.

43 papers.

Sort:

By area:

  1. 2023OmniControl: Control Any Joint at Any Time for Human Motion Generationcited by 238
  2. 2024Scaling Up Dynamic Human-Scene Interaction Modelingcited by 137
  3. 2023TLControl: Trajectory and Language Control for Human Motion Synthesiscited by 94
  4. 2023HOI-Diff: Text-Driven Synthesis of 3D Human-Object Interactions using Diffusion Modelscited by 93
  5. 2023EMDM: Efficient Motion Diffusion Model for Fast and High-Quality Motion Generationcited by 89
  6. 2025MotionStreamer: Streaming Motion Generation via Diffusion-based Autoregressive Model in Causal Latent Spacecited by 86
  7. 2024Rethinking Diffusion for Text-Driven Human Motion Generation: Redundant Representations, Evaluation, and Masked Autoregressioncited by 85
  8. 2024Generating Human Interaction Motions in Scenes with Text Controlcited by 80
  9. 2023PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interactioncited by 76
  10. 2024Large Motion Model for Unified Multi-Modal Motion Generationcited by 65
  11. 2024HOI-M3:Capture Multiple Humans and Objects Interaction within Contextual Environmentcited by 53
  12. 2025TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenizationcited by 51
31 more
  1. 2024Generating Human Motion in 3D Scenes from Text Descriptionscited by 49
  2. 2024THOR: Text to Human-Object Interaction Diffusion via Relation Interventioncited by 46
  3. 2024SMooDi: Stylized Motion Diffusion Modelcited by 37
  4. 2025Absolute Coordinates Make Motion Generation Easycited by 29
  5. 2025Ready-to-React: Online Reaction Policy for Two-Character Interaction Generationcited by 28
  6. 2024Scaling Large Motion Models with Million-Level Human Motionscited by 22
  7. 2024Stratified Avatar Generation from Sparse Observationscited by 21
  8. 2024TriDi: Trilateral Diffusion of 3D Humans, Objects, and Interactionscited by 20
  9. 2024SCENIC: Scene-aware Semantic Navigation with Instruction-guided Controlcited by 13
  10. 2025Human-Object Interaction via Automatically Designed VLM-Guided Motion Policycited by 12
  11. 2025CoDA: Coordinated Diffusion Noise Optimization for Whole-Body Manipulation of Articulated Objectscited by 12
  12. 2025SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarioscited by 12
  13. 2025HOI-Dyn: Learning Interaction Dynamics for Human-Object Motion Diffusioncited by 10
  14. 2024Motion-2-To-3: Leveraging 2D Motion Data for 3D Motion Generationscited by 9
  15. 2024You Think, You ACT: The New Task of Arbitrary Text to Motion Generationcited by 8
  16. 2023Ins-HOI: Instance Aware Human-Object Interactions Recoverycited by 7
  17. 2024FORCE: Physics-aware Human-object Interactioncited by 7
  18. 20253D Human Interaction Generation: A Surveycited by 7
  19. 2025TaxaDiffusion: Progressively Trained Diffusion Model for Fine-Grained Species Generationcited by 7
  20. 2024Two-Person Interaction Augmentation with Skeleton Priorscited by 5
  21. 2024CrowdMoGen: Zero-Shot Text-Driven Collective Motion Generationcited by 4
  22. 2024Decomposed Vector-Quantized Variational Autoencoder for Human Grasp Generationcited by 4
  23. 2025MATRIX: Mask Track Alignment for Interaction-aware Video Generationcited by 4
  24. 2024Agent-to-Sim: Learning Interactive Behavior Models from Casual Longitudinal Videoscited by 3
  25. 2025Spatial-Temporal Multi-Scale Quantization for Flexible Motion Generationcited by 3
  26. 2024CoNav: A Benchmark for Human-Centered Collaborative Navigationcited by 1
  27. 2025SyncMV4D: Synchronized Multi-view Joint Diffusion of Appearance and Motion for Hand-Object Interaction Synthesiscited by 1
  28. 2026FUSION: Full-Body Unified Motion Prior for Body and Hands via Diffusioncited by 1
  29. 2026Stability-Driven Motion Generation for Object-Guided Human-Human Co-Manipulationcited by 1
  30. 2026GIRAF: Towards Generalizable Human Interactions with Articulated Objectscited by 1
  31. 2026DeSeG: Decoupling Semantic Intent and Geometric Constraints for Physically Plausible Human-Scene Interactioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Controllable Human-Object Interaction Synthesis
  2. 2024Human-Aware 3D Scene Generation with Spatially-constrained Diffusion Models
  3. 2024Generation of Complex 3D Human Motion by Temporal and Spatial Composition of Diffusion Models
  4. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusion
  5. 2023AMD: Autoregressive Motion Diffusion
  6. 2025Object-Aware 4D Human Motion Generation
  7. 2025Strong and Controllable 3D Motion Generation
  8. 2025Multi-Person Interaction Generation from Two-Person Motion Priors