mlsanityData to 14 Oct 2026

Weakly-Supervised 3D Reconstruction of Clothed Humans via Normal Maps

arXiv 2311.16042 ↗First posted 27 Nov 2023cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

We present a novel deep learning-based approach to the 3D reconstruction of clothed humans using weak supervision via 2D normal maps. Given a single RGB image or multiview images, our network infers a signed distance function (SDF) discretized on a tetrahedral mesh surrounding the body in a rest pose. Subsequently, inferred pose and camera parameters are used to generate a normal map from the SDF. A key aspect of our approach is the use of Marching Tetrahedra to (uniquely) compute a triangulated surface from the SDF on the tetrahedral mesh, facilitating straightforward differentiation (and thus backpropagation). Thus, given only ground truth normal maps (with no volumetric information ground truth information), we can train the network to produce SDF values from corresponding RGB images. Optionally, an additional multiview loss leads to improved results. We demonstrate the efficacy of our approach for both network inference and 3D reconstruction.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 78 of its 105 references that are in the map. First posted on arXiv 27 Nov 2023.

Builds on

78 of 105 references.

References · 78

  1. 2020End-to-End Object Detection with Transformerscited by 7,414
  2. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
  3. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
  4. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,179
  5. 2017End-to-end Recovery of Human Shape and Posecited by 885
  6. 2019PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitizationcited by 818
  7. 2020Implicit Geometric Regularization for Learning Shapescited by 710
  8. 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 700
  9. 2020Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humanscited by 634
  10. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 620
  11. 2020PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitizationcited by 521
  12. 2023Neuralangelo: High-Fidelity Neural Surface Reconstructioncited by 467
38 more
  1. 2021Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesiscited by 460
  2. 2023Humans in 4D: Reconstructing and Tracking Humans with Transformerscited by 350
  3. 2021ICON: Implicit Clothed humans Obtained from Normalscited by 283
  4. 20193D Morphable Face Models -- Past, Present and Futurecited by 270
  5. 2022NeuMan: Neural Human Radiance Field from a Single Videocited by 244
  6. 2018Video Based Reconstruction of 3D People Modelscited by 238
  7. 2020PaMIR: Parametric Model-Conditioned Implicit Representation for Image-based Human Reconstructioncited by 231
  8. 2018Learning to Estimate 3D Human Pose and Shape from a Single Color Imagecited by 226
  9. 2022ECON: Explicit Clothed humans Optimized via Normal integrationcited by 199
  10. 2018Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose and Shape Estimationcited by 185
  11. 2019DeepHuman: 3D Human Reconstruction from a Single Imagecited by 182
  12. 2020DeepCap: Monocular Human Performance Capture Using Weak Supervisioncited by 162
  13. 2023HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motioncited by 161
  14. 2021Probabilistic Modeling for Human Mesh Recoverycited by 153
  15. 2022SelfRecon: Self Reconstruction Your Digital Avatar from Monocular Videocited by 144
  16. 2018BodyNet: Volumetric Inference of 3D Human Body Shapescited by 141
  17. 2022InstantAvatar: Learning Avatars from Monocular Video in 60 Secondscited by 138
  18. 2023Vid2Avatar: 3D Avatar Reconstruction from Videos in the Wild via Self-supervised Scene Decompositioncited by 134
  19. 2020MeshSDF: Differentiable Iso-Surface Extractioncited by 112
  20. 2021HumanNeRF: Efficiently Generated Human Radiance Field from Sparse Inputscited by 112
  21. 2022Recovering 3D Human Mesh from Monocular Images: A Surveycited by 103
  22. 2021Neural-GIF: Neural Generalized Implicit Functions for Animating People in Clothingcited by 102
  23. 2020Weakly Supervised 3D Human Pose and Shape Reconstruction with Normalizing Flowscited by 94
  24. 2018LiveCap: Real-time Human Performance Capture from Monocular Videocited by 93
  25. 2021The Power of Points for Modeling Humans in Clothingcited by 90
  26. 20183D Human Pose Estimation in the Wild by Adversarial Learningcited by 86
  27. 2023MonoHuman: Animatable Human Neural Field from Monocular Videocited by 85
  28. 2023AvatarReX: Real-time Expressive Full-body Avatarscited by 84
  29. 2022Capturing and Animation of Body and Clothing from Monocular Videocited by 83
  30. 2021MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Imagescited by 78
  31. 2022PermutoSDF: Fast Multi-View Reconstruction with Implicit Surfaces using Permutohedral Latticescited by 73
  32. 2022gDNA: Towards Generative Detailed Neural Avatarscited by 72
  33. 2023High-fidelity 3D Human Digitization from Single 2K Resolution Imagescited by 70
  34. 2018SiCloPe: Silhouette-Based Clothed Peoplecited by 68
  35. 2019Moulding Humans: Non-parametric 3D Human Shape Estimation from Single Imagescited by 67
  36. 2021Modeling Clothing as a Separate Layer for an Animatable Human Avatarcited by 66
  37. 2023X-Avatar: Expressive Human Avatarscited by 66
  38. 2022Neural Surface Reconstruction of Dynamic Scenes with Monocular RGB-D Cameracited by 57

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024A Neural-Network-Based Approach for Loose-Fitting Clothingcited by 2