mlsanityData to 7 Oct 2026

Scene-aware Generative Network for Human Motion Synthesis

arXiv 2105.14804 ↗First posted 31 May 2021cs.CVcited by 70

Not on the map. Why?

Abstract (arXiv)

We revisit human motion synthesis, a task useful in various real world applications, in this paper. Whereas a number of methods have been developed previously for this task, they are often limited in two aspects: focusing on the poses while leaving the location movement behind, and ignoring the impact of the environment on the human motion. In this paper, we propose a new framework, with the interaction between the scene and the human motion taken into account. Considering the uncertainty of human motion, we formulate this task as a generative task, whose objective is to generate plausible human motion conditioned on both the scene and the human initial position. This framework factorizes the distribution of human motions into a distribution of movement trajectories conditioned on scenes and that of body pose dynamics conditioned on both scenes and trajectories. We further derive a GAN based learning approach, with discriminators to enforce the compatibility between the human motion and the contextual scene as well as the 3D to 2D projection constraints. We assess the effectiveness of the proposed method on two challenging datasets, which cover both synthetic and real world environments.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Builds on 29 of its 37 references that are in the map. First posted on arXiv 31 May 2021.

Builds on

29 of 37 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  5. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  6. 2017Improved Training of Wasserstein GANscited by 2,223
  7. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  8. 2015Spatial Transformer Networkscited by 1,423
  9. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,199
  10. 2018BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentationcited by 645
  11. 2020Action2Motion: Conditioned Generation of 3D Human Motionscited by 438
  12. 2016Deeper Depth Prediction with Fully Convolutional Residual Networkscited by 402
17 more
  1. 2017On human motion prediction using recurrent neural networkscited by 336
  2. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 294
  3. 2019Learning Trajectory Dependencies for Human Motion Predictioncited by 245
  4. 2020Long-term Human Motion Prediction with Scene Contextcited by 191
  5. 2017HP-GAN: Probabilistic 3D human motion prediction via GANcited by 176
  6. 2020Dynamic Multiscale Graph Neural Networks for 3D Skeleton-Based Human Motion Predictioncited by 132
  7. 2019Generating 3D People in Scenes without Peoplecited by 127
  8. 2020Motion Guided 3D Pose Estimation from Videoscited by 85
  9. 2019Putting Humans in a Scene: Learning Affordance in 3D Indoor Environmentscited by 72
  10. 2020FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integrationcited by 72
  11. 2018Pose Guided Human Video Generationcited by 67
  12. 2017Deep Video Generation, Prediction and Completion of Human Action Sequencescited by 50
  13. 2019Delving Deep Into Hybrid Annotations for 3D Human Recovery in the Wildcited by 38
  14. 2020Malleable 2.5D Convolution: Learning Receptive Fields along the Depth-axis for RGB-D Scene Parsingcited by 24
  15. 2020PointINS: Point-based Instance Segmentationcited by 17
  16. 2020Chasing the Tail in Monocular 3D Human Reconstruction with Prototype Memorycited by 11
  17. 2018Sequential Context Encoding for Duplicate Removalcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 6; 2023: 21; 2024: 19; 2025: 16; 2026: 2.

70 papers.

Sort:

By area:

  1. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 237
  2. 2023Object Motion Guided Human Motion Synthesiscited by 206
  3. 2023InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactionscited by 205
  4. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 181
  5. 2021FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integrationcited by 168
  6. 2022HUMANISE: Language-conditioned Human Motion Generation in 3D Scenescited by 156
  7. 2023CG-HOI: Contact-Guided 3D Human-Object Interaction Generationcited by 137
  8. 2024Scaling Up Dynamic Human-Scene Interaction Modelingcited by 137
  9. 2024MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpaintingcited by 101
  10. 2023Human Motion Generation: A Surveycited by 95
  11. 2023TLControl: Trajectory and Language Control for Human Motion Synthesiscited by 93
  12. 2022Towards Diverse and Natural Scene-aware 3D Human Motion Synthesiscited by 91
38 more
  1. 2024Generating Human Interaction Motions in Scenes with Text Controlcited by 80
  2. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordancecited by 79
  3. 2023NIFTY: Neural Object Interaction Fields for Guided Human Motion Synthesiscited by 78
  4. 2024Seamless Human Motion Composition with Blended Positional Encodingscited by 77
  5. 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 63
  6. 2024DiffH2O: Diffusion-Based Synthesis of Hand-Object Interactions from Textual Descriptionscited by 60
  7. 2024Autonomous Character-Scene Interaction Synthesis from Text Instructioncited by 56
  8. 2024ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactionscited by 54
  9. 2023ArtiGrasp: Physically Plausible Synthesis of Bi-Manual Dexterous Grasping and Articulationcited by 53
  10. 2023Interactive Character Control with Auto-Regressive Motion Diffusion Modelscited by 52
  11. 2023Physically Plausible Full-Body Hand-Object Interaction Synthesiscited by 52
  12. 2023Locomotion-Action-Manipulation: Synthesizing Human-Scene Interactions in Complex 3D Environmentscited by 51
  13. 2025TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenizationcited by 46
  14. 2022MultiAct: Long-Term 3D Human Motion Generation from Multiple Action Labelscited by 45
  15. 2023Hierarchical Generation of Human-Object Interactions with Diffusion Probabilistic Modelscited by 43
  16. 2023Generating Continual Human Motion in Diverse 3D Scenescited by 39
  17. 2024AnySkill: Learning Open-Vocabulary Physical Skill for Interactive Agentscited by 32
  18. 2021Monocular 3D Reconstruction of Interacting Hands via Collision-Aware Factorized Refinementscited by 27
  19. 2022Contact-aware Human Motion Forecastingcited by 27
  20. 2022H3WB: Human3.6M 3D WholeBody Dataset and Benchmarkcited by 23
  21. 2021Not All Voxels Are Equal: Semantic Scene Completion from the Point-Voxel Perspectivecited by 22
  22. 2023InterControl: Zero-shot Human Interaction Generation by Controlling Every Jointcited by 22
  23. 2023Dynamic Storyboard Generation in an Engine-based Virtual Environment for Video Productioncited by 21
  24. 2021VoteHMR: Occlusion-Aware Voting Network for Robust 3D Human Mesh Recovery from Partial Point Cloudscited by 18
  25. 2023OmniMotionGPT: Animal Motion Generation with Limited Datacited by 18
  26. 2024TriDi: Trilateral Diffusion of 3D Humans, Objects, and Interactionscited by 18
  27. 2025UniEgoMotion: A Unified Model for Egocentric Motion Reconstruction, Forecasting, and Generationcited by 17
  28. 2024SyncDiff: Synchronized Motion Diffusion for Multi-Body Human-Object Interaction Synthesiscited by 12
  29. 2025SceneMI: Motion In-betweening for Modeling Human-Scene Interactionscited by 11
  30. 2025MOSPA: Human Motion Generation Driven by Spatial Audiocited by 11
  31. 2026InterPrior: Scaling Generative Control for Physics-Based Human-Object Interactionscited by 10
  32. 2021Dyadic Human Motion Predictioncited by 9
  33. 2024Move-in-2D: 2D-Conditioned Human Motion Generationcited by 8
  34. 2023What Happened 3 Seconds Ago? Inferring the Past with Thermal Imagingcited by 7
  35. 2023Scene-aware Human Motion Forecasting via Mutual Distance Predictioncited by 7
  36. 2025Target-Aware Video Diffusion Modelscited by 7
  37. 2025Ponimator: Unfolding Interactive Pose for Versatile Human-human Interaction Animationcited by 7
  38. 2024Multimodal Sense-Informed Prediction of 3D Human Motionscited by 6

50 most cited of 70 All 70 (JSON)