mlsanityData to 14 Oct 2026

Scaling Up Dynamic Human-Scene Interaction Modeling

arXiv 2403.08629 ↗First posted 13 Mar 2024cs.CVcited by 137

Not on the map. Why?

Abstract (arXiv)

Confronting the challenges of data scarcity and advanced motion synthesis in human-scene interaction modeling, we introduce the TRUMANS dataset alongside a novel HSI motion synthesis method. TRUMANS stands as the most comprehensive motion-captured HSI dataset currently available, encompassing over 15 hours of human interactions across 100 indoor scenes. It intricately captures whole-body human motions and part-level object dynamics, focusing on the realism of contact. This dataset is further scaled up by transforming physical environments into exact virtual models and applying extensive augmentations to appearance and motion for both humans and objects while maintaining interaction fidelity. Utilizing TRUMANS, we devise a diffusion-based autoregressive model that efficiently generates HSI sequences of any length, taking into account both scene context and intended actions. In experiments, our approach shows remarkable zero-shot generalizability on a range of 3D scene datasets (e.g., PROX, Replica, ScanNet, ScanNet++), producing motions that closely mimic original motion-captured sequences, as confirmed by quantitative experiments and human studies.

Key facts. Cited by 137 papers in the map (data through 14 Oct 2026). Builds on 51 of its 67 references that are in the map. First posted on arXiv 13 Mar 2024.

Builds on

51 of 67 references.

References · 51

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  4. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  5. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  6. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
  7. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,214
  8. 2022Human Motion Diffusion Modelcited by 1,080
  9. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 904
  10. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 764
  11. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 588
  12. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 405
38 more
  1. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 380
  2. 2023Human Motion Diffusion as a Generative Priorcited by 357
  3. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 295
  4. 2020I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Imagecited by 274
  5. 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 249
  6. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 239
  7. 2023BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motioncited by 236
  8. 2023Guided Motion Diffusion for Controllable Human Motion Synthesiscited by 223
  9. 2023Object Motion Guided Human Motion Synthesiscited by 212
  10. 2021Stochastic Scene-Aware Motion Predictioncited by 203
  11. 2020Long-term Human Motion Prediction with Scene Contextcited by 192
  12. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 182
  13. 2022Capturing and Inferring Dense Full-Body Human-Scene Contactcited by 180
  14. 2020Populating 3D Scenes by Learning Human-Scene Interactioncited by 164
  15. 2022HUMANISE: Language-conditioned Human Motion Generation in 3D Scenescited by 158
  16. 2021GOAL: Generating 4D Whole-Body Motion for Hand-Object Graspingcited by 144
  17. 2021Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensorscited by 137
  18. 2020Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenescited by 135
  19. 2022COUCH: Towards Controllable Human-Chair Interactionscited by 135
  20. 2019Generating 3D People in Scenes without Peoplecited by 127
  21. 2022IMos: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactionscited by 124
  22. 2023Synthesizing Diverse Human Motions in 3D Indoor Scenescited by 109
  23. 2023Unified Human-Scene Interaction via Prompted Chain-of-Contactscited by 94
  24. 2022Towards Diverse and Natural Scene-aware 3D Human Motion Synthesiscited by 91
  25. 2019Context-aware Human Motion Predictioncited by 84
  26. 2022Compositional Human-Scene Interaction Synthesis with Semantic Controlcited by 83
  27. 2023CIRCLE: Capture In Rich Contextual Environmentscited by 79
  28. 2022Full-Body Articulated Human-Object Interactioncited by 78
  29. 2019Putting Humans in a Scene: Learning Affordance in 3D Indoor Environmentscited by 72
  30. 2021Scene-aware Generative Network for Human Motion Synthesiscited by 70
  31. 2023DECO: Dense Estimation of 3D Human-Scene Contact In The Wildcited by 54
  32. 2023Locomotion-Action-Manipulation: Synthesizing Human-Scene Interactions in Complex 3D Environmentscited by 51
  33. 2023Synthesizing Physically Plausible Human Motions in 3D Scenescited by 49
  34. 20233D Human Mesh Estimation from Virtual Markerscited by 44
  35. 2023Hierarchical Generation of Human-Object Interactions with Diffusion Probabilistic Modelscited by 43
  36. 2023Generating Continual Human Motion in Diverse 3D Scenescited by 39
  37. 2022Interaction Replica: Tracking Human-Object Interaction and Scene Changes From Human Motioncited by 27
  38. 2022Contact-aware Human Motion Forecastingcited by 27

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 30; 2025: 61; 2026: 46.

137 papers.

Sort:

By area:

  1. 2024Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wildcited by 114
  2. 2024Rethinking Diffusion for Text-Driven Human Motion Generation: Redundant Representations, Evaluation, and Masked Autoregressioncited by 85
  3. 2024DartControl: A Diffusion-Based Autoregressive Motion Model for Real-Time Text-Driven Motion Controlcited by 84
  4. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordancecited by 80
  5. 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 72
  6. 20243DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generationcited by 71
  7. 2025Go to Zero: Towards Zero-shot Motion Generation with Million-scale Datacited by 68
  8. 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 64
  9. 2024Human-Object Interaction from Human-Level Instructionscited by 64
  10. 2024Autonomous Character-Scene Interaction Synthesis from Text Instructioncited by 56
  11. 2024HOI-M3:Capture Multiple Humans and Objects Interaction within Contextual Environmentcited by 53
  12. 2025TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenizationcited by 51
38 more
  1. 2024The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motioncited by 46
  2. 2024PhyRecon: Physically Plausible Neural Scene Reconstructioncited by 45
  3. 2024CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangementcited by 44
  4. 2025InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generationcited by 32
  5. 2025Decompositional Neural Scene Reconstruction with Generative Diffusion Priorcited by 31
  6. 2025HUMOTO: A 4D Dataset of Mocap Human Object Interactionscited by 31
  7. 2024SkillMimic: Learning Basketball Interaction Skills from Demonstrationscited by 30
  8. 2025Joint Optimization for 4D Human-Scene Reconstruction in the Wildcited by 29
  9. 2025Absolute Coordinates Make Motion Generation Easycited by 29
  10. 2024EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Viewscited by 26
  11. 2024ZeroHSI: Zero-Shot 4D Human-Scene Interaction by Video Generationcited by 25
  12. 2025A Survey on Human Interaction Motion Generationcited by 25
  13. 2025SkillMimic-V2: Learning Robust and Generalizable Interaction Skills from Sparse and Noisy Demonstrationscited by 25
  14. 2024F-HOI: Toward Fine-grained Semantic-Aligned 3D Human-Object Interactionscited by 24
  15. 2025The Quest for Generalizable Motion Generation: Data, Model, and Evaluationcited by 22
  16. 2024TriDi: Trilateral Diffusion of 3D Humans, Objects, and Interactionscited by 20
  17. 2025ChainHOI: Joint-based Kinematic Chain Modeling for Human-Object Interaction Generationcited by 20
  18. 2025Dexterous World Modelscited by 18
  19. 2025UniEgoMotion: A Unified Model for Egocentric Motion Reconstruction, Forecasting, and Generationcited by 17
  20. 2024Mimicking-Bench: A Benchmark for Generalizable Humanoid-Scene Interaction Learning via Human Mimickingcited by 16
  21. 2025SemGeoMo: Dynamic Contextual Human Motion Generation with Semantic and Geometric Guidancecited by 16
  22. 2025Dynamic Motion Blending for Versatile Motion Editingcited by 15
  23. 2024Sitcom-Crafter: A Plot-Driven Human Motion Generation System in 3D Scenescited by 14
  24. 2025PRIMAL: Physically Reactive and Interactive Motor Model for Avatar Learningcited by 14
  25. 2025HunyuanVideo-HOMA: Generic Human-Object Interaction in Multimodal Driven Human Animationcited by 14
  26. 2025Reconstructing 4D Spatial Intelligence: A Surveycited by 14
  27. 2026EgoSim: Egocentric World Simulator for Embodied Interaction Generationcited by 14
  28. 2024SCENIC: Scene-aware Semantic Navigation with Instruction-guided Controlcited by 13
  29. 2025InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsingcited by 13
  30. 2024SIMS: Simulating Stylized Human-Scene Interactions with Retrieval-Augmented Script Generationcited by 12
  31. 2025Human-Object Interaction via Automatically Designed VLM-Guided Motion Policycited by 12
  32. 2025CoDA: Coordinated Diffusion Noise Optimization for Whole-Body Manipulation of Articulated Objectscited by 12
  33. 2025SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarioscited by 12
  34. 2026SimRecon: SimReady Compositional Scene Reconstruction from Real Videoscited by 12
  35. 2024Lifting Motion to the 3D World via 2D Diffusioncited by 11
  36. 2024EnvPoser: Environment-aware Realistic Human Motion Estimation from Sparse Observations with Uncertainty Modelingcited by 11
  37. 2025SceneMI: Motion In-betweening for Modeling Human-Scene Interactionscited by 11
  38. 2025Shape My Moves: Text-Driven Shape-Aware Synthesis of Human Motionscited by 11

50 most cited of 137 All 137 (JSON)