mlsanityData to 7 Oct 2026

HUMANISE: Language-conditioned Human Motion Generation in 3D Scenes

arXiv 2210.09729 ↗First posted 18 Oct 2022cs.CV cs.AIcited by 156

Not on the map. Why?

Cited by 27 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Learning to generate diverse scene-aware and goal-oriented human motions in 3D scenes remains challenging due to the mediocre characteristics of the existing datasets on Human-Scene Interaction (HSI); they only have limited scale/quality and lack semantics. To fill in the gap, we propose a large-scale and semantic-rich synthetic HSI dataset, denoted as HUMANISE, by aligning the captured human motion sequences with various 3D indoor scenes. We automatically annotate the aligned motions with language descriptions that depict the action and the unique interacting objects in the scene; e.g., sit on the armchair near the desk. HUMANISE thus enables a new generation task, language-conditioned human motion generation in 3D scenes. The proposed task is challenging as it requires joint modeling of the 3D scene, human motion, and natural language. To tackle this task, we present a novel scene-and-language conditioned generative model that can produce 3D human motions of the desirable action interacting with the specified objects. Our experiments demonstrate that our model generates diverse and semantically consistent human motions in 3D scenes.

Key facts. Cited by 156 papers in the map (data through 7 Oct 2026). Builds on 46 of its 59 references that are in the map. First posted on arXiv 18 Oct 2022.

Builds on

46 of 59 references.

References · 46

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  4. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  5. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  6. 2020Point Transformercited by 1,462
  7. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  8. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  9. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  10. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 885
  11. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 497
  12. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
34 more
  1. 2020Action2Motion: Conditioned Generation of 3D Human Motionscited by 438
  2. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 397
  3. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  4. 2017Embodied Question Answeringcited by 335
  5. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 294
  6. 2021BABEL: Bodies, Action and Behavior with English Labelscited by 261
  7. 2019Language2Pose: Natural Language Grounded Pose Forecastingcited by 257
  8. 2021Stochastic Scene-Aware Motion Predictioncited by 203
  9. 2020Long-term Human Motion Prediction with Scene Contextcited by 191
  10. 2020DLow: Diversifying Latent Flows for Diverse Human Motion Predictioncited by 184
  11. 2021Synthesis of Compositional Animations from Textual Descriptionscited by 177
  12. 2017HP-GAN: Probabilistic 3D human motion prediction via GANcited by 176
  13. 2022Capturing and Inferring Dense Full-Body Human-Scene Contactcited by 176
  14. 2020Populating 3D Scenes by Learning Human-Scene Interactioncited by 164
  15. 2020Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenescited by 135
  16. 2019Generating 3D People in Scenes without Peoplecited by 127
  17. 2018Convolutional Sequence to Sequence Model for Human Dynamicscited by 115
  18. 2020We are More than Our Joints: Predicting how 3D Bodies Movecited by 115
  19. 2019Look, Listen, and Act: Towards Audio-Visual Embodied Navigationcited by 95
  20. 2021LanguageRefer: Spatial-Language Model for 3D Visual Groundingcited by 88
  21. 2019Holistic++ Scene Understanding: Single-view 3D Holistic Scene Parsing and Human Pose Estimation with Human-Object Interaction and Physical Commonsensecited by 71
  22. 2021Scene-aware Generative Network for Human Motion Synthesiscited by 70
  23. 2021Physics-based Human Motion Estimation and Synthesis from Videoscited by 65
  24. 2022Human-Aware Object Placement for Visual Environment Reconstructioncited by 56
  25. 2020LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activitiescited by 46
  26. 2018Human Motion Modeling using DVGANscited by 45
  27. 2021Language Grounding with 3D Objectscited by 43
  28. 20213D Question Answeringcited by 41
  29. 2019Learning to Sit: Synthesizing Human-Chair Interactions via Hierarchical Controlcited by 39
  30. 2021YouRefIt: Embodied Reference Understanding with Language and Gesturecited by 32
  31. 2019SplitNet: Sim2Sim and Task2Task Transfer for Embodied Visual Navigationcited by 29
  32. 2021VLGrammar: Grounded Grammar Induction of Vision and Languagecited by 22
  33. 2022Understanding Embodied Reference with Touch-Line Transformercited by 16
  34. 2020PLACE: Proximity Learning of Articulation and Contact in 3D Environmentscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 26; 2024: 46; 2025: 58; 2026: 24.

156 papers.

Sort:

By area:

  1. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 237
  2. 2023OmniControl: Control Any Joint at Any Time for Human Motion Generationcited by 233
  3. 2023Object Motion Guided Human Motion Synthesiscited by 206
  4. 2023InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactionscited by 205
  5. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 181
  6. 2024MotionLCM: Real-time Controllable Motion Generation via Latent Consistency Modelcited by 164
  7. 2023CG-HOI: Contact-Guided 3D Human-Object Interaction Generationcited by 137
  8. 2024Scaling Up Dynamic Human-Scene Interaction Modelingcited by 137
  9. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  10. 2023Controllable Human-Object Interaction Synthesiscited by 107
  11. 2023HumanTOMATO: Text-aligned Whole-body Motion Generationcited by 103
  12. 2023Inter-X: Towards Versatile Human-Human Interaction Analysiscited by 98
38 more
  1. 2023Human Motion Generation: A Surveycited by 95
  2. 2023Unified Human-Scene Interaction via Prompted Chain-of-Contactscited by 91
  3. 2023HOI-Diff: Text-Driven Synthesis of 3D Human-Object Interactions using Diffusion Modelscited by 89
  4. 2024Multi-Track Timeline Control for Text-Driven 3D Human Motion Generationcited by 88
  5. 2024Generating Human Interaction Motions in Scenes with Text Controlcited by 80
  6. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordancecited by 79
  7. 2023NIFTY: Neural Object Interaction Fields for Guided Human Motion Synthesiscited by 78
  8. 2022Full-Body Articulated Human-Object Interactioncited by 76
  9. 2023ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenescited by 68
  10. 2025Go to Zero: Towards Zero-shot Motion Generation with Million-scale Datacited by 67
  11. 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 63
  12. 2023SINC: Spatial Composition of 3D Human Motions for Simultaneous Action Generationcited by 62
  13. 2024Human-Object Interaction from Human-Level Instructionscited by 62
  14. 2024ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactionscited by 54
  15. 2024HOI-M3:Capture Multiple Humans and Objects Interaction within Contextual Environmentcited by 52
  16. 2024ScaMo: Exploring the Scaling Law in Autoregressive Motion Generation Modelcited by 52
  17. 2023Locomotion-Action-Manipulation: Synthesizing Human-Scene Interactions in Complex 3D Environmentscited by 51
  18. 2024Generating Human Motion in 3D Scenes from Text Descriptionscited by 49
  19. 2024THOR: Text to Human-Object Interaction Diffusion via Relation Interventioncited by 46
  20. 2024PhyRecon: Physically Plausible Neural Scene Reconstructioncited by 44
  21. 2025GENMO: A GENeralist Model for Human MOtioncited by 43
  22. 2023I'M HOI: Inertia-aware Monocular Capture of 3D Human-Object Interactionscited by 42
  23. 2023GenZI: Zero-Shot 3D Human-Scene Interaction Generationcited by 40
  24. 2024CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangementcited by 40
  25. 2024HIMO: A New Benchmark for Full-Body Human Interacting with Multiple Objectscited by 34
  26. 2024LaserHuman: Language-guided Scene-aware Human Motion Generation in Free Environmentcited by 33
  27. 2024HMD^2: Environment-aware Motion Generation from Single Egocentric Head-Mounted Devicecited by 26
  28. 2024ZeroHSI: Zero-Shot 4D Human-Scene Interaction by Video Generationcited by 25
  29. 2025A Survey on Human Interaction Motion Generationcited by 25
  30. 20223D Segmentation of Humans in Point Clouds with Synthetic Datacited by 24
  31. 2023Revisit Human-Scene Interaction via Space Occupancycited by 24
  32. 2024F-HOI: Toward Fine-grained Semantic-Aligned 3D Human-Object Interactionscited by 24
  33. 20253D Scene Generation: A Surveycited by 24
  34. 2023InterControl: Zero-shot Human Interaction Generation by Controlling Every Jointcited by 22
  35. 2025ChainHOI: Joint-based Kinematic Chain Modeling for Human-Object Interaction Generationcited by 19
  36. 2023OmniMotionGPT: Animal Motion Generation with Limited Datacited by 18
  37. 2023Language-guided Human Motion Synthesis with Atomic Actionscited by 17
  38. 2024Pay Attention and Move Better: Harnessing Attention for Interactive Motion Generation and Training-free Editingcited by 17

50 most cited of 156 All 156 (JSON)