mlsanityData to 14 Oct 2026

Generating Human Interaction Motions in Scenes with Text Control

arXiv 2404.10685 ↗First posted 16 Apr 2024cs.CV cs.GRcited by 80

On the map: Diffusion models → Video generation

Abstract (arXiv)

We present TeSMo, a method for text-controlled scene-aware motion generation based on denoising diffusion models. Previous text-to-motion methods focus on characters in isolation without considering scenes due to the limited availability of datasets that include motion, text descriptions, and interactive scenes. Our approach begins with pre-training a scene-agnostic text-to-motion diffusion model, emphasizing goal-reaching constraints on large-scale motion-capture datasets. We then enhance this model with a scene-aware component, fine-tuned using data augmented with detailed scene information, including ground plane and object shapes. To facilitate training, we embed annotated navigation and interaction motions within scenes. The proposed method produces realistic and diverse human-object interactions, such as navigation and sitting, in different scenes with various object shapes, orientations, initial body positions, and poses. Extensive experiments demonstrate that our approach surpasses prior techniques in terms of the plausibility of human-scene interactions, as well as the realism and variety of the generated motions. Code will be released upon publication of this work at https://research.nvidia.com/labs/toronto-ai/tesmo.

Key facts. Cited by 80 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 42 of its 52 references that are in the map. First posted on arXiv 16 Apr 2024.

Builds on

42 of 52 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023HOI-Diff: Text-Driven Synthesis of 3D Human-Object Interactions using Diffusion Models11 Dec
  2. 2023Controllable Human-Object Interaction Synthesis6 Dec
  3. 2023Hierarchical Generation of Human-Object Interactions with Diffusion Probabilistic Models3 Oct
  4. 2023NIFTY: Neural Object Interaction Fields for Guided Human Motion Synthesis14 Jul
  5. 2023Guided Motion Diffusion for Controllable Human Motion Synthesis21 May
  6. 2023Human Motion Diffusion as a Generative Prior2 Mar
  7. 2022Human Motion Diffusion Model29 Sep
  8. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model31 Aug
  9. 2022Video Diffusion Models7 Apr

Other references · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  4. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,236
  5. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 380
  6. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 295
  7. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 278
  8. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 239
  9. 2023OmniControl: Control Any Joint at Any Time for Human Motion Generationcited by 238
  10. 2023Object Motion Guided Human Motion Synthesiscited by 212
  11. 2021Stochastic Scene-Aware Motion Predictioncited by 203
  12. 2022HUMANISE: Language-conditioned Human Motion Generation in 3D Scenescited by 158
21 more
  1. 2021GOAL: Generating 4D Whole-Body Motion for Hand-Object Graspingcited by 144
  2. 2023Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusioncited by 138
  3. 2023CG-HOI: Contact-Guided 3D Human-Object Interaction Generationcited by 138
  4. 2020Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenescited by 135
  5. 2022COUCH: Towards Controllable Human-Chair Interactionscited by 135
  6. 2023Synthesizing Diverse Human Motions in 3D Indoor Scenescited by 109
  7. 2023Synthesizing Physical Character-Scene Interactionscited by 104
  8. 2019Efficient Learning on Point Clouds with Basis Point Setscited by 103
  9. 2023Human Motion Generation: A Surveycited by 96
  10. 2023Unified Human-Scene Interaction via Prompted Chain-of-Contactscited by 94
  11. 2024Multi-Track Timeline Control for Text-Driven 3D Human Motion Generationcited by 91
  12. 2019Context-aware Human Motion Predictioncited by 84
  13. 2022Compositional Human-Scene Interaction Synthesis with Semantic Controlcited by 83
  14. 2021Scene-aware Generative Network for Human Motion Synthesiscited by 70
  15. 2021The Wanderings of Odysseus in 3D Scenescited by 59
  16. 2022MIME: Human-Aware 3D Scene Generationcited by 57
  17. 2022Human-Aware Object Placement for Visual Environment Reconstructioncited by 56
  18. 2023Locomotion-Action-Manipulation: Synthesizing Human-Scene Interactions in Complex 3D Environmentscited by 51
  19. 2019Learning to Sit: Synthesizing Human-Chair Interactions via Hierarchical Controlcited by 40
  20. 2023ROAM: Robust and Object-Aware Motion Generation Using Neural Pose Descriptorscited by 23
  21. 2023Scene Synthesis from Human Motioncited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 16; 2025: 48; 2026: 16.

80 papers.

Sort:

By area:

  1. 2024CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character controlcited by 109
  2. 2024Human-Object Interaction from Human-Level Instructionscited by 64
  3. 2024Autonomous Character-Scene Interaction Synthesis from Text Instructioncited by 56
  4. 2025TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenizationcited by 51
  5. 2025GENMO: A GENeralist Model for Human MOtioncited by 49
  6. 2024CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangementcited by 44
  7. 2024SMooDi: Stylized Motion Diffusion Modelcited by 37
  8. 2024Unimotion: Unifying 3D Human Motion Synthesis and Understandingcited by 30
  9. 2024BimArt: A Unified Approach for the Synthesis of 3D Bimanual Interaction with Articulated Objectscited by 30
  10. 2025Absolute Coordinates Make Motion Generation Easycited by 29
  11. 2025PARC: Physics-based Augmentation with Reinforcement Learning for Character Controllerscited by 28
  12. 2025A Survey on Human Interaction Motion Generationcited by 25
38 more
  1. 2024TriDi: Trilateral Diffusion of 3D Humans, Objects, and Interactionscited by 20
  2. 2025ChainHOI: Joint-based Kinematic Chain Modeling for Human-Object Interaction Generationcited by 20
  3. 2025Sketch2Anim: Towards Transferring Sketch Storyboards into 3D Animationcited by 17
  4. 2025UniEgoMotion: A Unified Model for Egocentric Motion Reconstruction, Forecasting, and Generationcited by 17
  5. 2025SemGeoMo: Dynamic Contextual Human Motion Generation with Semantic and Geometric Guidancecited by 16
  6. 2024SyncDiff: Synchronized Motion Diffusion for Multi-Body Human-Object Interaction Synthesiscited by 14
  7. 2025SocialGen: Modeling Multi-Human Social Interaction with Language Modelscited by 14
  8. 2024SCENIC: Scene-aware Semantic Navigation with Instruction-guided Controlcited by 13
  9. 2025InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsingcited by 13
  10. 2025EgoTwin: Dreaming Body and View in First Personcited by 13
  11. 2024SIMS: Simulating Stylized Human-Scene Interactions with Retrieval-Augmented Script Generationcited by 12
  12. 2025Human-Object Interaction via Automatically Designed VLM-Guided Motion Policycited by 12
  13. 2025CoDA: Coordinated Diffusion Noise Optimization for Whole-Body Manipulation of Articulated Objectscited by 12
  14. 2025SceneMI: Motion In-betweening for Modeling Human-Scene Interactionscited by 11
  15. 2024Motion-2-To-3: Leveraging 2D Motion Data for 3D Motion Generationscited by 9
  16. 2025Motion Synthesis with Sparse and Flexible Keyjoint Controlcited by 9
  17. 2025Multi-Person Interaction Generation from Two-Person Motion Priorscited by 8
  18. 20253D Human Interaction Generation: A Surveycited by 7
  19. 2025Target-Aware Video Diffusion Modelscited by 7
  20. 2025GenHOI: Generalizing Text-driven 4D Human-Object Interaction Synthesis for Unseen Objectscited by 7
  21. 2025ByteLoom: Weaving Geometry-Consistent Human-Object Interactions through Progressive Curriculum Learningcited by 7
  22. 2024Diffusion Implicit Policy for Unpaired Scene-aware Motion Synthesiscited by 6
  23. 2025GenHSI: Controllable Generation of Human-Scene Interaction Videoscited by 6
  24. 2025Unified Multi-Modal Interactive & Reactive 3D Motion Generation via Rectified Flowcited by 6
  25. 2025HOSIG: Full-Body Human-Object-Scene Interaction Generation with Hierarchical Scene Perceptioncited by 5
  26. 2025InterAgent: Physics-based Multi-agent Command Execution via Diffusion on Interaction Graphscited by 5
  27. 2024Learning to Generate Diverse Pedestrian Movements from Web Videos with Noisy Labelscited by 4
  28. 2025DAViD: Modeling Dynamic Affordance of 3D Objects Using Pre-trained Video Diffusion Modelscited by 4
  29. 2025FlowMotion: Target-Predictive Conditional Flow Matching for Jitter-Reduced Text-Driven Human Motion Generationcited by 4
  30. 2025Event-Driven Storytelling with Multiple Lifelike Humans in a 3D Scenecited by 4
  31. 2025SimGenHOI: Physically Realistic Whole-Body Humanoid-Object Interaction via Generative Modeling and Reinforcement Learningcited by 4
  32. 2025MDD: A Dataset for Text-and-Music Conditioned Duet Dance Generationcited by 4
  33. 2025Uni-Inter: Unifying 3D Human Motion Synthesis Across Diverse Interaction Contextscited by 4
  34. 2025VHOI: Controllable Video Generation of Human-Object Interactions from Sparse Trajectories via Motion Densificationcited by 4
  35. 2024MM-Conv: A Multi-modal Conversational Dataset for Virtual Humanscited by 3
  36. 2025Versatile Physics-based Character Control with Hybrid Latent Representationcited by 3
  37. 2025SIGHT: Synthesizing Image-Text Conditioned and Geometry-Guided 3D Hand-Object Trajectoriescited by 3
  38. 2025Spatial-Temporal Multi-Scale Quantization for Flexible Motion Generationcited by 3

50 most cited of 80 All 80 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Generating Human Motion in 3D Scenes from Text Descriptions
  2. 2025UniHM: Universal Human Motion Generation with Object Interactions in Indoor Scenes
  3. 2024Animate Your Motion: Turning Still Images into Dynamic Videos
  4. 2025Auto-Regressive Diffusion for Generating 3D Human-Object Interactions
  5. 2024LaserHuman: Language-guided Scene-aware Human Motion Generation in Free Environment
  6. 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction
  7. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordance
  8. 2023Enhanced Fine-grained Motion Diffusion for Text-driven Human Motion Synthesis