# Scene Synthesis from Human Motion

arXiv: https://arxiv.org/abs/2301.01424 · HTML: https://mlsanity.com/paper/2301.01424/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 51 references that are in the map.
- First posted on arXiv 4 Jan 2023.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image](https://mlsanity.com/paper/1607.08128/) (2016)
- [VirtualHome: Simulating Household Activities via Programs](https://mlsanity.com/paper/1806.07011/) (2018)
- [3D-FRONT: 3D Furnished Rooms with layOuts and semaNTics](https://mlsanity.com/paper/2011.09127/) (2020)
- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [LSTM: A Search Space Odyssey](https://mlsanity.com/paper/1503.04069/) (2015)
- [3D-FUTURE: 3D Furniture shape with TextURE](https://mlsanity.com/paper/2009.09633/) (2020)
- [Resolving 3D Human Pose Ambiguities with 3D Scene Constraints](https://mlsanity.com/paper/1908.06963/) (2019)
- [BEHAVE: Dataset and Method for Tracking Human Object Interactions](https://mlsanity.com/paper/2204.06950/) (2022)
- [Robust Motion In-betweening](https://mlsanity.com/paper/2102.04942/) (2021)
- [Stochastic Scene-Aware Motion Prediction](https://mlsanity.com/paper/2108.08284/) (2021)
- [ATISS: Autoregressive Transformers for Indoor Scene Synthesis](https://mlsanity.com/paper/2110.03675/) (2021)
- [Populating 3D Scenes by Learning Human-Scene Interaction](https://mlsanity.com/paper/2012.11581/) (2020)
- [SceneFormer: Indoor Scene Generation with Transformers](https://mlsanity.com/paper/2012.09793/) (2020)
- [Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenes](https://mlsanity.com/paper/2012.05522/) (2020)
- [Generating 3D People in Scenes without People](https://mlsanity.com/paper/1912.02923/) (2019)
- [Fast and Flexible Indoor Scene Synthesis via Deep Convolutional Generative Models](https://mlsanity.com/paper/1811.12463/) (2018)
- [Learning Motion Priors for 4D Human Body Capture in 3D Scenes](https://mlsanity.com/paper/2108.10399/) (2021)
- [SpiralNet++: A Fast and Highly Efficient Mesh Convolution Operator](https://mlsanity.com/paper/1911.05856/) (2019)
- [GIMO: Gaze-Informed Human Motion Prediction in Context](https://mlsanity.com/paper/2204.09443/) (2022)
- [Learning to Act Properly: Predicting and Explaining Affordances from Images](https://mlsanity.com/paper/1712.07576/) (2017)
- [Putting Humans in a Scene: Learning Affordance in 3D Indoor Environments](https://mlsanity.com/paper/1903.05690/) (2019)
- [Holistic++ Scene Understanding: Single-view 3D Holistic Scene Parsing and Human Pose Estimation with Human-Object Interaction and Physical Commonsense](https://mlsanity.com/paper/1909.01507/) (2019)
- [Human-Aware Object Placement for Visual Environment Reconstruction](https://mlsanity.com/paper/2203.03609/) (2022)
- [SceneGraphNet: Neural Message Passing for 3D Indoor Scene Augmentation](https://mlsanity.com/paper/1907.11308/) (2019)
- [End-to-End Optimization of Scene Layout](https://mlsanity.com/paper/2007.11744/) (2020)
- [Binge Watching: Scaling Affordance Learning from Sitcoms](https://mlsanity.com/paper/1804.03080/) (2018)
- [Hierarchical Style-based Networks for Motion Synthesis](https://mlsanity.com/paper/2008.10162/) (2020)

## Built on by (most cited first, 20 of 20)

- [Generating Human Interaction Motions in Scenes with Text Control](https://mlsanity.com/paper/2404.10685/) (2024)
- [NIFTY: Neural Object Interaction Fields for Guided Human Motion Synthesis](https://mlsanity.com/paper/2307.07511/) (2023)
- [Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clips](https://mlsanity.com/paper/2309.05663/) (2023)
- [ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactions](https://mlsanity.com/paper/2401.10232/) (2024)
- [LEMON: Learning 3D Human-Object Interaction Relation from 2D Images](https://mlsanity.com/paper/2312.08963/) (2023)
- [THOR: Text to Human-Object Interaction Diffusion via Relation Intervention](https://mlsanity.com/paper/2403.11208/) (2024)
- [Revisit Human-Scene Interaction via Space Occupancy](https://mlsanity.com/paper/2312.02700/) (2023)
- [3D Scene Generation: A Survey](https://mlsanity.com/paper/2505.05474/) (2025)
- [ROAM: Robust and Object-Aware Motion Generation Using Neural Pose Descriptors](https://mlsanity.com/paper/2308.12969/) (2023)
- [TriDi: Trilateral Diffusion of 3D Humans, Objects, and Interactions](https://mlsanity.com/paper/2412.06334/) (2024)
- [Language-driven Scene Synthesis using Multi-conditional Diffusion Model](https://mlsanity.com/paper/2310.15948/) (2023)
- [Physics-based Scene Layout Generation from Human Motion](https://mlsanity.com/paper/2405.12460/) (2024)
- [Ins-HOI: Instance Aware Human-Object Interactions Recovery](https://mlsanity.com/paper/2312.09641/) (2023)
- [ReSpace: Text-Driven Autoregressive 3D Indoor Scene Synthesis and Editing](https://mlsanity.com/paper/2506.02459/) (2025)
- [CRAFT: Designing Creative and Functional 3D Objects](https://mlsanity.com/paper/2412.03889/) (2024)
- [Seeing Without Eyes: 4D Human-Scene Understanding from Wearable IMUs](https://mlsanity.com/paper/2604.21926/) (2026)
- [Diverse 3D Human Pose Generation in Scenes based on Decoupled Structure](https://mlsanity.com/paper/2406.05691/) (2024)
- [AccioScene: Compositional 3D Scene Generation via Graph Diffusion and Interaction-driven Critics](https://mlsanity.com/paper/2502.06819/) (2025)
- [SceneTeract: Probing and Improving Agent-Aware Activity Reasoning in 3D Indoor Scenes](https://mlsanity.com/paper/2603.29798/) (2026)
- [Efficient Human-Contact Representation for Human-Scene Interaction](https://mlsanity.com/paper/2608.09388/) (2026)

## Abstract (arXiv)

Large-scale capture of human motion with diverse, complex scenes, while immensely useful, is often considered prohibitively costly. Meanwhile, human motion alone contains rich information about the scene they reside in and interact with. For example, a sitting human suggests the existence of a chair, and their leg position further implies the chair's pose. In this paper, we propose to synthesize diverse, semantically reasonable, and physically plausible scenes based on human motion. Our framework, Scene Synthesis from HUMan MotiON (SUMMON), includes two steps. It first uses ContactFormer, our newly introduced contact predictor, to obtain temporally consistent contact labels from human motion. Based on these predictions, SUMMON then chooses interacting objects and optimizes physical plausibility losses; it further populates the scene with objects that do not interact with humans. Experimental results demonstrate that SUMMON synthesizes feasible, plausible, and diverse scenes and has the potential to generate extensive human-scene interaction data for the community.