mlsanityData to 7 Oct 2026

Single-image coherent reconstruction of objects and humans

arXiv 2408.08086 ↗First posted 15 Aug 2024cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Existing methods for reconstructing objects and humans from a monocular image suffer from severe mesh collisions and performance limitations for interacting occluding objects. This paper introduces a method to obtain a globally consistent 3D reconstruction of interacting objects and people from a single image. Our contributions include: 1) an optimization framework, featuring a collision loss, tailored to handle human-object and human-human interactions, ensuring spatially coherent scene reconstruction; and 2) a novel technique to robustly estimate 6 degrees of freedom (DOF) poses, specifically for heavily occluded objects, exploiting image inpainting. Notably, our proposed method operates effectively on images from real-world scenarios, without necessitating scene or object-level 3D supervision. Extensive qualitative and quantitative evaluation against existing methods demonstrates a significant reduction in collisions in the final reconstructions of scenes with multiple interacting humans and objects and a more coherent scene reconstruction.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 66 of its 90 references that are in the map. First posted on arXiv 15 Aug 2024.

Builds on

66 of 90 references.

References · 66

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  4. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  5. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  6. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
  7. 2017End-to-end Recovery of Human Shape and Posecited by 883
  8. 2019PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitizationcited by 818
  9. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
  10. 2018Pixel2Mesh: Generating 3D Mesh Models from Single RGB Imagescited by 649
  11. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 625
  12. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
38 more
  1. 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 587
  2. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  3. 2018AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generationcited by 539
  4. 2020PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitizationcited by 521
  5. 2019PointRend: Image Segmentation as Renderingcited by 439
  6. 2017Neural 3D Mesh Renderercited by 437
  7. 2019Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoningcited by 420
  8. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 397
  9. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  10. 2018Learning Category-Specific Mesh Reconstruction from Image Collectionscited by 317
  11. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 294
  12. 2019Convolutional Mesh Regression for Single-Image Human Shape Reconstructioncited by 270
  13. 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 246
  14. 2018Video Based Reconstruction of 3D People Modelscited by 237
  15. 2018Learning to Estimate 3D Human Pose and Shape from a Single Color Imagecited by 226
  16. 2020Monocular, One-stage, Regression of Multiple 3D Peoplecited by 216
  17. 2018Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose and Shape Estimationcited by 185
  18. 2020ContactPose: A Dataset of Grasps with Object Contact and Hand Posecited by 180
  19. 2020Grasping Field: Learning Implicit Representations for Human Graspscited by 178
  20. 2016Learning a Predictable and Generative Vector Representation for Objectscited by 173
  21. 2019Learning to Reconstruct People in Clothing from a Single RGB Cameracited by 168
  22. 2021FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integrationcited by 168
  23. 2020Populating 3D Scenes by Learning Human-Scene Interactioncited by 164
  24. 2019EdgeConnect: Generative Image Inpainting with Adversarial Edge Learningcited by 161
  25. 2020Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wildcited by 152
  26. 2019ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imagingcited by 145
  27. 20203D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humanscited by 141
  28. 2021Putting People in their Place: Monocular Regression of 3D People in Depthcited by 141
  29. 2020Coherent Reconstruction of Multiple Humans from a Single Imagecited by 136
  30. 2022COUCH: Towards Controllable Human-Chair Interactionscited by 134
  31. 2023One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformercited by 128
  32. 2020Self-supervised Single-view 3D Reconstruction via Semantic Consistencycited by 127
  33. 2018Detailed Human Avatars from Monocular Videocited by 109
  34. 2019Deep Mesh Reconstruction from Single RGB Images via Topology Modification Networkscited by 106
  35. 2017Self-supervised Learning of Motion Capturecited by 97
  36. 2021On Self-Contact and Human Posecited by 88
  37. 2019ContactGrasp: Functional Multi-finger Grasp Synthesis from Contactcited by 87
  38. 2019Holistic++ Scene Understanding: Single-view 3D Holistic Scene Parsing and Human Pose Estimation with Human-Object Interaction and Physical Commonsensecited by 71

50 most cited of 66 All 66 (JSON)

Built on by

Not cited yet.