mlsanityData to 14 Oct 2026

Locomotion-Action-Manipulation: Synthesizing Human-Scene Interactions in Complex 3D Environments

arXiv 2301.02667 ↗First posted 9 Jan 2023cs.CV cs.GR cs.ROcited by 51

Not on the map. Why?

Abstract (arXiv)

Synthesizing interaction-involved human motions has been challenging due to the high complexity of 3D environments and the diversity of possible human behaviors within. We present LAMA, Locomotion-Action-MAnipulation, to synthesize natural and plausible long-term human movements in complex indoor environments. The key motivation of LAMA is to build a unified framework to encompass a series of everyday motions including locomotion, scene interaction, and object manipulation. Unlike existing methods that require motion data "paired" with scanned 3D scenes for supervision, we formulate the problem as a test-time optimization by using human motion capture data only for synthesis. LAMA leverages a reinforcement learning framework coupled with a motion matching algorithm for optimization, and further exploits a motion editing framework via manifold learning to cover possible variations in interaction and manipulation. Throughout extensive experiments, we demonstrate that LAMA outperforms previous approaches in synthesizing realistic motions in various challenging scenarios. Project page: https://jiyewise.github.io/projects/LAMA/ .

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Builds on 39 of its 79 references that are in the map. First posted on arXiv 9 Jan 2023.

Builds on

39 of 79 references.

References · 39

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  5. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,214
  6. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 784
  7. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 595
  8. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 501
  9. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 485
  10. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 405
  11. 2017On human motion prediction using recurrent neural networkscited by 338
  12. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 295
27 more
  1. 2021Robust Motion In-betweeningcited by 229
  2. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 206
  3. 2021Stochastic Scene-Aware Motion Predictioncited by 203
  4. 2020Long-term Human Motion Prediction with Scene Contextcited by 192
  5. 2021Character Controllers Using Motion VAEscited by 184
  6. 2022Capturing and Inferring Dense Full-Body Human-Scene Contactcited by 180
  7. 2020Populating 3D Scenes by Learning Human-Scene Interactioncited by 164
  8. 2022HUMANISE: Language-conditioned Human Motion Generation in 3D Scenescited by 158
  9. 2020Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wildcited by 152
  10. 2021GOAL: Generating 4D Whole-Body Motion for Hand-Object Graspingcited by 144
  11. 2020Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenescited by 135
  12. 2022COUCH: Towards Controllable Human-Chair Interactionscited by 135
  13. 2019Generating 3D People in Scenes without Peoplecited by 127
  14. 2022Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motioncited by 126
  15. 2020Skeleton-Aware Networks for Deep Motion Retargetingcited by 124
  16. 2022CHORE: Contact, Human and Object REconstruction from a single RGB imagecited by 95
  17. 2020Unpaired Motion Style Transfer from Video to Animationcited by 94
  18. 2021SAGA: Stochastic Whole-Body Grasping with Contactcited by 91
  19. 2022Towards Diverse and Natural Scene-aware 3D Human Motion Synthesiscited by 91
  20. 2017Learning to Generate Long-term Future via Hierarchical Predictioncited by 88
  21. 2022Compositional Human-Scene Interaction Synthesis with Semantic Controlcited by 83
  22. 2021Scene-aware Generative Network for Human Motion Synthesiscited by 70
  23. 2022TOCH: Spatio-Temporal Object-to-Hand Correspondence for Motion Refinementcited by 70
  24. 2022NeuralHOFusion: Neural Volumetric Rendering under Human-object Interactionscited by 44
  25. 2019Learning to Sit: Synthesizing Human-Chair Interactions via Hierarchical Controlcited by 40
  26. 2021D3D-HOI: Dynamic 3D Human-Object Interactions from Videoscited by 39
  27. 2022Learning to Use Chopsticks in Diverse Gripping Stylescited by 29

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 12; 2024: 17; 2025: 16; 2026: 6.

51 papers.

Sort:

By area:

  1. 2023Object Motion Guided Human Motion Synthesiscited by 212
  2. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 182
  3. 2023CG-HOI: Contact-Guided 3D Human-Object Interaction Generationcited by 138
  4. 2024Scaling Up Dynamic Human-Scene Interaction Modelingcited by 137
  5. 2024Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wildcited by 114
  6. 2023Controllable Human-Object Interaction Synthesiscited by 109
  7. 2024Generating Human Interaction Motions in Scenes with Text Controlcited by 80
  8. 2023PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interactioncited by 76
  9. 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 72
  10. 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 64
  11. 2024DiffH2O: Diffusion-Based Synthesis of Hand-Object Interactions from Textual Descriptionscited by 62
  12. 2024ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactionscited by 57
38 more
  1. 2023CAMS: CAnonicalized Manipulation Spaces for Category-Level Functional Hand-Object Manipulation Synthesiscited by 56
  2. 2024Autonomous Character-Scene Interaction Synthesis from Text Instructioncited by 56
  3. 2023ArtiGrasp: Physically Plausible Synthesis of Bi-Manual Dexterous Grasping and Articulationcited by 55
  4. 2023Physically Plausible Full-Body Hand-Object Interaction Synthesiscited by 52
  5. 2025TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenizationcited by 51
  6. 2023Synthesizing Physically Plausible Human Motions in 3D Scenescited by 49
  7. 2024AnySkill: Learning Open-Vocabulary Physical Skill for Interactive Agentscited by 32
  8. 2025InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generationcited by 32
  9. 2024Mocap Everyone Everywhere: Lightweight Motion Capture With Smartwatches and a Head-Mounted Cameracited by 31
  10. 2024HMD^2: Environment-aware Motion Generation from Single Egocentric Head-Mounted Devicecited by 27
  11. 2023Revisit Human-Scene Interaction via Space Occupancycited by 25
  12. 2024ZeroHSI: Zero-Shot 4D Human-Scene Interaction by Video Generationcited by 25
  13. 2025A Survey on Human Interaction Motion Generationcited by 25
  14. 2023ROAM: Robust and Object-Aware Motion Generation Using Neural Pose Descriptorscited by 23
  15. 2024Survey on Modeling of Human-made Articulated Objectscited by 14
  16. 2024SCENIC: Scene-aware Semantic Navigation with Instruction-guided Controlcited by 13
  17. 2025Human-Object Interaction via Automatically Designed VLM-Guided Motion Policycited by 12
  18. 2025Learning to Transfer Human Hand Skills for Robot Manipulationscited by 11
  19. 2025SceneMI: Motion In-betweening for Modeling Human-Scene Interactionscited by 11
  20. 2025Motion Synthesis with Sparse and Flexible Keyjoint Controlcited by 9
  21. 2024FORCE: Physics-aware Human-object Interactioncited by 7
  22. 2025Target-Aware Video Diffusion Modelscited by 7
  23. 2025EgoReAct: Egocentric Video-Driven 3D Human Reaction Generationcited by 7
  24. 2024Diffusion Implicit Policy for Unpaired Scene-aware Motion Synthesiscited by 6
  25. 2025HOI-PAGE: Zero-Shot Human-Object Interaction Generation with Part Affordance Guidancecited by 6
  26. 2025DAViD: Modeling Dynamic Affordance of 3D Objects Using Pre-trained Video Diffusion Modelscited by 4
  27. 2024COLLAGE: Collaborative Human-Agent Interaction Generation using Hierarchical Latent Diffusion and Language Modelscited by 3
  28. 2024Agent-to-Sim: Learning Interactive Behavior Models from Casual Longitudinal Videoscited by 3
  29. 2025Decoupled Generative Modeling for Human-Object Interaction Synthesiscited by 3
  30. 2026InfBaGel: Human-Object-Scene Interaction Generation with Dynamic Perception and Iterative Refinementcited by 3
  31. 2025MonSTeR: a Unified Model for Motion, Scene, Text Retrievalcited by 2
  32. 2026Seeing Without Eyes: 4D Human-Scene Understanding from Wearable IMUscited by 2
  33. 2025H2OFlow: Grounding Human-Object Affordances with 3D Generative Models and Dense Diffused Flowscited by 1
  34. 2025Human Motion Synthesis in 3D Scenes via Unified Scene Semantic Occupancycited by 1
  35. 2026MOCHI: Motion Enhancement of Collaborative Human-object Interactionscited by 1
  36. 2026GIRAF: Towards Generalizable Human Interactions with Articulated Objectscited by 1
  37. 2023Personalized Pose Forecastingcited by 0
  38. 2026SynAgent: Generalizable Cooperative Humanoid Manipulation via Solo-to-Cooperative Agent Synergycited by 0

50 most cited of 51 All 51 (JSON)