# Pose2Room: Understanding 3D Scenes from Human Activities

arXiv: https://arxiv.org/abs/2112.03030 · HTML: https://mlsanity.com/paper/2112.03030/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 81 references that are in the map.
- First posted on arXiv 1 Dec 2021.

## Builds on (most cited first, 30 of 35)

- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling](https://mlsanity.com/paper/1610.07584/) (2016)
- [VirtualHome: Simulating Household Activities via Programs](https://mlsanity.com/paper/1806.07011/) (2018)
- [Vector Neurons: A General Framework for SO(3)-Equivariant Networks](https://mlsanity.com/paper/2104.12229/) (2021)
- [Resolving 3D Human Pose Ambiguities with 3D Scene Constraints](https://mlsanity.com/paper/1908.06963/) (2019)
- [Stochastic Scene-Aware Motion Prediction](https://mlsanity.com/paper/2108.08284/) (2021)
- [Long-term Human Motion Prediction with Scene Context](https://mlsanity.com/paper/2007.03672/) (2020)
- [Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Image](https://mlsanity.com/paper/2002.12212/) (2020)
- [History Repeats Itself: Human Motion Prediction via Motion Attention](https://mlsanity.com/paper/2007.11755/) (2020)
- [Populating 3D Scenes by Learning Human-Scene Interaction](https://mlsanity.com/paper/2012.11581/) (2020)
- [Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors](https://mlsanity.com/paper/2103.17265/) (2021)
- [Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenes](https://mlsanity.com/paper/2012.05522/) (2020)
- [Generating 3D People in Scenes without People](https://mlsanity.com/paper/1912.02923/) (2019)
- [BundleFusion: Real-time Globally Consistent 3D Reconstruction using On-the-fly Surface Re-integration](https://mlsanity.com/paper/1604.01093/) (2016)
- [Sparse Inertial Poser: Automatic 3D Human Pose Estimation from Sparse IMUs](https://mlsanity.com/paper/1703.08014/) (2017)
- [3D AffordanceNet: A Benchmark for Visual Object Affordance Understanding](https://mlsanity.com/paper/2103.16397/) (2021)
- [Holistic 3D Scene Understanding from a Single Image with Implicit Representation](https://mlsanity.com/paper/2103.06422/) (2021)
- [Context-aware Human Motion Prediction](https://mlsanity.com/paper/1904.03419/) (2019)
- [Panoptic 3D Scene Reconstruction From a Single RGB Image](https://mlsanity.com/paper/2111.02444/) (2021)
- [Holistic 3D Scene Parsing and Reconstruction from a Single RGB Image](https://mlsanity.com/paper/1808.02201/) (2018)
- [Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrieve](https://mlsanity.com/paper/2007.13034/) (2020)
- [Multimodal Shape Completion via Conditional Generative Adversarial Networks](https://mlsanity.com/paper/2003.07717/) (2020)
- [FroDO: From Detections to 3D Objects](https://mlsanity.com/paper/2005.05125/) (2020)
- [Deep Inertial Poser: Learning to Reconstruct Human Pose from Sparse Inertial Measurements in Real Time](https://mlsanity.com/paper/1810.04703/) (2018)
- [CoReNet: Coherent 3D scene reconstruction from a single RGB image](https://mlsanity.com/paper/2004.12989/) (2020)
- [Learning to Sit: Synthesizing Human-Chair Interactions via Hierarchical Control](https://mlsanity.com/paper/1908.07423/) (2019)

## Built on by (most cited first, 22 of 22)

- [COUCH: Towards Controllable Human-Chair Interactions](https://mlsanity.com/paper/2205.00541/) (2022)
- [Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clips](https://mlsanity.com/paper/2309.05663/) (2023)
- [MIME: Human-Aware 3D Scene Generation](https://mlsanity.com/paper/2212.04360/) (2022)
- [ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactions](https://mlsanity.com/paper/2401.10232/) (2024)
- [Object pop-up: Can we infer 3D objects and their poses from human interactions alone?](https://mlsanity.com/paper/2306.00777/) (2023)
- [Revisit Human-Scene Interaction via Space Occupancy](https://mlsanity.com/paper/2312.02700/) (2023)
- [Learning 3D Scene Priors with 2D Supervision](https://mlsanity.com/paper/2211.14157/) (2022)
- [Scene Synthesis from Human Motion](https://mlsanity.com/paper/2301.01424/) (2023)
- [OptiScene: LLM-driven Indoor Scene Layout Generation via Scaled Human-aligned Data Synthesis and Multi-Stage Preference Optimization](https://mlsanity.com/paper/2506.07570/) (2025)
- [PhySIC: Physically Plausible 3D Human-Scene Interaction and Contact from a Single Image](https://mlsanity.com/paper/2510.11649/) (2025)
- [Language-driven Scene Synthesis using Multi-conditional Diffusion Model](https://mlsanity.com/paper/2310.15948/) (2023)
- [Human-Aware 3D Scene Generation with Spatially-constrained Diffusion Models](https://mlsanity.com/paper/2406.18159/) (2024)
- [Cloth2Body: Generating 3D Human Body Mesh from 2D Clothing](https://mlsanity.com/paper/2309.16189/) (2023)
- [S-INF: Towards Realistic Indoor Scene Synthesis via Scene Implicit Neural Field](https://mlsanity.com/paper/2412.17561/) (2024)
- [Geometric Visual Fusion Graph Neural Networks for Multi-Person Human-Object Interaction Recognition in Videos](https://mlsanity.com/paper/2506.03440/) (2025)
- [Non-contact Multimodal Indoor Human Monitoring Systems: A Survey](https://mlsanity.com/paper/2312.07601/) (2023)
- [Seeing Without Eyes: 4D Human-Scene Understanding from Wearable IMUs](https://mlsanity.com/paper/2604.21926/) (2026)
- [Scene and Human in One World: Reconstruction in a Feedforward Pass](https://mlsanity.com/paper/2606.27720/) (2026)
- [AccioScene: Compositional 3D Scene Generation via Graph Diffusion and Interaction-driven Critics](https://mlsanity.com/paper/2502.06819/) (2025)
- [uLayout: Unified Room Layout Estimation for Perspective and Panoramic Images](https://mlsanity.com/paper/2503.21562/) (2025)
- [CSA-Graphs: A Privacy-Preserving Structural Dataset for Child Sexual Abuse Research](https://mlsanity.com/paper/2604.07132/) (2026)
- [Efficient Human-Contact Representation for Human-Scene Interaction](https://mlsanity.com/paper/2608.09388/) (2026)

## Abstract (arXiv)

With wearable IMU sensors, one can estimate human poses from wearable devices without requiring visual input~citevon2017sparse. In this work, we pose the question: Can we reason about object structure in real-world environments solely from human trajectory information? Crucially, we observe that human motion and interactions tend to give strong information about the objects in a scene -- for instance a person sitting indicates the likely presence of a chair or sofa. To this end, we propose P2R-Net to learn a probabilistic 3D model of the objects in a scene characterized by their class categories and oriented 3D bounding boxes, based on an input observed human trajectory in the environment. P2R-Net models the probability distribution of object class as well as a deep Gaussian mixture model for object boxes, enabling sampling of multiple, diverse, likely modes of object configurations from an observed human trajectory. In our experiments we show that P2R-Net can effectively learn multi-modal distributions of likely objects for human motions, and produce a variety of plausible object structures of the environment, even without any visual information. The results demonstrate that P2R-Net consistently outperforms the baselines on the PROX dataset and the VirtualHome platform.