mlsanityData to 7 Oct 2026

Human-Aware Object Placement for Visual Environment Reconstruction

arXiv 2203.03609 ↗First posted 7 Mar 2022cs.CV cs.AI cs.GRcited by 56

Not on the map. Why?

Abstract (arXiv)

Humans are in constant contact with the world as they move through it and interact with it. This contact is a vital source of information for understanding 3D humans, 3D scenes, and the interactions between them. In fact, we demonstrate that these human-scene interactions (HSIs) can be leveraged to improve the 3D reconstruction of a scene from a monocular RGB video. Our key idea is that, as a person moves through a scene and interacts with it, we accumulate HSIs across multiple input images, and optimize the 3D scene to reconstruct a consistent, physically plausible and functional 3D scene layout. Our optimization-based approach exploits three types of HSI constraints: (1) humans that move in a scene are occluded or occlude objects, thus, defining the depth ordering of the objects, (2) humans move through free space and do not interpenetrate objects, (3) when humans and objects are in contact, the contact surfaces occupy the same place in space. Using these constraints in an optimization formulation across all observations, we significantly improve the 3D scene layout reconstruction. Furthermore, we show that our scene reconstruction can be used to refine the initial 3D human pose and shape (HPS) estimation. We evaluate the 3D scene layout reconstruction and HPS estimation qualitatively and quantitatively using the PROX and PiGraphs datasets. The code and data are available for research purposes at https://mover.is.tue.mpg.de/.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Builds on 74 of its 105 references that are in the map. First posted on arXiv 7 Mar 2022.

Builds on

74 of 105 references.

References · 74

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Mask R-CNNcited by 7,259
  3. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  4. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  5. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  6. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  7. 2017End-to-end Recovery of Human Shape and Posecited by 883
  8. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  9. 2016Unsupervised Monocular Depth Estimation with Left-Right Consistencycited by 819
  10. 2019PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitizationcited by 818
  11. 2018Deep Ordinal Regression Network for Monocular Depth Estimationcited by 721
  12. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
38 more
  1. 2018Pixel2Mesh: Generating 3D Mesh Models from Single RGB Imagescited by 649
  2. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 625
  3. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
  4. 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 587
  5. 2018AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generationcited by 539
  6. 2020PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitizationcited by 521
  7. 2019PointRend: Image Segmentation as Renderingcited by 439
  8. 2016Deeper Depth Prediction with Fully Convolutional Residual Networkscited by 402
  9. 2017A simple yet effective baseline for 3d human pose estimationcited by 401
  10. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 397
  11. 2021PARE: Part Attention Regressor for 3D Human Body Estimationcited by 334
  12. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 320
  13. 2021HuMoR: 3D Human Motion Model for Robust Pose Estimationcited by 319
  14. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 294
  15. 2021ICON: Implicit Clothed humans Obtained from Normalscited by 283
  16. 2019Convolutional Mesh Regression for Single-Image Human Shape Reconstructioncited by 270
  17. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 251
  18. 2018Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodiescited by 239
  19. 2019DeepHuman: 3D Human Reconstruction from a Single Imagecited by 182
  20. 2019DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracingcited by 179
  21. 2020Whole-Body Human Pose Estimation in the Wildcited by 178
  22. 2017PoseTrack: A Benchmark for Human Pose Estimation and Trackingcited by 173
  23. 2020Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Imagecited by 170
  24. 2020Populating 3D Scenes by Learning Human-Scene Interactioncited by 164
  25. 2018Monocular Total Capture: Posing Face, Body, and Hands in the Wildcited by 162
  26. 2021Hand-Object Contact Consistency Reasoning for Human Grasps Generationcited by 160
  27. 2020Monocular Expressive Body Regression through Body-Driven Attentioncited by 155
  28. 2020Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wildcited by 152
  29. 2021GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Camerascited by 151
  30. 2020Reconstructing Hand-Object Interactions in the Wildcited by 143
  31. 2021GOAL: Generating 4D Whole-Body Motion for Hand-Object Graspingcited by 142
  32. 2018BodyNet: Volumetric Inference of 3D Human Body Shapescited by 141
  33. 2021Putting People in their Place: Monocular Regression of 3D People in Depthcited by 141
  34. 2021Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Timecited by 140
  35. 2020Coherent Reconstruction of Multiple Humans from a Single Imagecited by 136
  36. 2021End-to-End Human Object Interaction Detection with HOI Transformercited by 134
  37. 2019Generating 3D People in Scenes without Peoplecited by 127
  38. 2020CPF: Learning a Contact Potential Field to Model the Hand-Object Interactioncited by 127

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 13; 2023: 17; 2024: 14; 2025: 5; 2026: 7.

56 papers.

Sort:

By area:

  1. 2022Generating Holistic 3D Human Motion from Speechcited by 180
  2. 2022Capturing and Inferring Dense Full-Body Human-Scene Contactcited by 176
  3. 2022HUMANISE: Language-conditioned Human Motion Generation in 3D Scenescited by 156
  4. 2022COUCH: Towards Controllable Human-Chair Interactionscited by 134
  5. 2023TeCH: Text-guided Reconstruction of Lifelike Clothed Humanscited by 104
  6. 2022CHORE: Contact, Human and Object REconstruction from a single RGB imagecited by 95
  7. 2022InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interactioncited by 90
  8. 2024Generating Human Interaction Motions in Scenes with Text Controlcited by 80
  9. 2022TOCH: Spatio-Temporal Object-to-Hand Correspondence for Motion Refinementcited by 69
  10. 2022NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactionscited by 67
  11. 2023Visibility Aware Human-Object Interaction Tracking from Single RGB Cameracited by 58
  12. 2023EgoHumans: An Egocentric 3D Multi-Human Benchmarkcited by 58
38 more
  1. 2022MIME: Human-Aware 3D Scene Generationcited by 57
  2. 2022Occluded Human Mesh Recoverycited by 52
  3. 2024HOI-M3:Capture Multiple Humans and Objects Interaction within Contextual Environmentcited by 52
  4. 2023SLOPER4D: A Scene-Aware Dataset for Global 4D Human Pose Estimation in Urban Environmentscited by 47
  5. 2024THOR: Text to Human-Object Interaction Diffusion via Relation Interventioncited by 46
  6. 2023Human Pose as Compositional Tokenscited by 45
  7. 2023I'M HOI: Inertia-aware Monocular Capture of 3D Human-Object Interactionscited by 42
  8. 2023Template Free Reconstruction of Human-object Interaction with Procedural Interaction Generationcited by 37
  9. 2023Object pop-up: Can we infer 3D objects and their poses from human interactions alone?cited by 36
  10. 2023PACE: Human and Camera Motion Estimation from in-the-wild Videoscited by 33
  11. 2024Beyond the Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained 2D Diffusion Modelscited by 32
  12. 2024Harmony4D: A Video Dataset for In-The-Wild Close Human Interactionscited by 31
  13. 2022Reconstructing Action-Conditioned Human-Object Interactions Using Commonsense Knowledge Priorscited by 30
  14. 2022Scene-aware Egocentric 3D Human Pose Estimationcited by 30
  15. 2023Revisit Human-Scene Interaction via Space Occupancycited by 24
  16. 2023Scene Synthesis from Human Motioncited by 20
  17. 2023Scene-Aware 3D Multi-Human Motion Capture from a Single Cameracited by 19
  18. 2023CHORUS: Learning Canonicalized 3D Human-Object Spatial Relations from Unbounded Synthesized Imagescited by 19
  19. 2024TriDi: Trilateral Diffusion of 3D Humans, Objects, and Interactionscited by 18
  20. 2025PhySIC: Physically Plausible 3D Human-Scene Interaction and Contact from a Single Imagecited by 13
  21. 2024RHOBIN Challenge: Reconstruction of Human Object Interactioncited by 10
  22. 2024Physics-based Scene Layout Generation from Human Motioncited by 10
  23. 2023ProxyCap: Real-time Monocular Full-body Capture in World Space via Human-Centric Proxy-to-Motion Learningcited by 9
  24. 2025ScoreHOI: Physically Plausible Reconstruction of Human-Object Interaction via Score-Guided Diffusioncited by 9
  25. 2025Zero-Shot Human-Object Interaction Synthesis with Multimodal Priorscited by 7
  26. 2024ROOT: VLM based System for Indoor Scene Understanding and Beyondcited by 6
  27. 2024Two-Person Interaction Augmentation with Skeleton Priorscited by 5
  28. 2023GraMMaR: Ground-aware Motion Model for 3D Human Motion Reconstructioncited by 4
  29. 2023Cloth2Body: Generating 3D Human Body Mesh from 2D Clothingcited by 3
  30. 2025InteracTalker: Prompt-Based Human-Object Interaction with Co-Speech Gesture Generationcited by 3
  31. 2023Physically Plausible 3D Human-Scene Reconstruction from Monocular RGB Image using an Adversarial Learning Approachcited by 2
  32. 2024TFS-NeRF: Template-Free NeRF for Semantic 3D Reconstruction of Dynamic Scenecited by 2
  33. 2024Diverse 3D Human Pose Generation in Scenes based on Decoupled Structurecited by 1
  34. 2026GRAFT: Geometric Refinement and Fitting Transformer for Human Scene Reconstructioncited by 1
  35. 2026Seeing Without Eyes: 4D Human-Scene Understanding from Wearable IMUscited by 1
  36. 2026Puppeteer: Object-Grounded Posture-Aware Co-Speech Gesture Generationcited by 1
  37. 2022Unified Simulation, Perception, and Generation of Human Behaviorcited by 0
  38. 2024Single-image coherent reconstruction of objects and humanscited by 0

50 most cited of 56 All 56 (JSON)