mlsanityData to 7 Oct 2026

Fully Understanding Generic Objects: Modeling, Segmentation, and Reconstruction

arXiv 2104.00858 ↗First posted 2 Apr 2021cs.CVcited by 16

Not on the map. Why?

Abstract (arXiv)

Inferring 3D structure of a generic object from a 2D image is a long-standing objective of computer vision. Conventional approaches either learn completely from CAD-generated synthetic data, which have difficulty in inference from real images, or generate 2.5D depth image via intrinsic decomposition, which is limited compared to the full 3D reconstruction. One fundamental challenge lies in how to leverage numerous real 2D images without any 3D ground truth. To address this issue, we take an alternative approach with semi-supervised learning. That is, for a 2D image of a generic object, we decompose it into latent representations of category, shape and albedo, lighting and camera projection matrix, decode the representations to segmented 3D shape and albedo respectively, and fuse these components to render an image well approximating the input image. Using a category-adaptive 3D joint occupancy field (JOF), we show that the complete shape and albedo modeling enables us to leverage real 2D images in both modeling and model fitting. The effectiveness of our approach is demonstrated through superior 3D reconstruction from a single image, being either synthetic or real, and shape segmentation.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 57 of its 76 references that are in the map. First posted on arXiv 2 Apr 2021.

Builds on

57 of 76 references.

References · 57

  1. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  2. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  3. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  4. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  5. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  6. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  7. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  8. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
  9. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  10. 2019PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitizationcited by 818
  11. 2020Convolutional Occupancy Networkscited by 760
  12. 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 699
38 more
  1. 2018Pixel2Mesh: Generating 3D Mesh Models from Single RGB Imagescited by 649
  2. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
  3. 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 587
  4. 2017Neural 3D Mesh Renderercited by 437
  5. 2020U2-Net: Going Deeper with Nested U-Structure for Salient Object Detectioncited by 410
  6. 2020Implicit Functions in Feature Space for 3D Shape Reconstruction and Completioncited by 343
  7. 2019DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstructioncited by 332
  8. 2018Learning Category-Specific Mesh Reconstruction from Image Collectionscited by 317
  9. 2019What Do Single-view 3D Reconstruction Networks Learn?cited by 259
  10. 2019M3D-RPN: Monocular 3D Region Proposal Network for Object Detectioncited by 251
  11. 2018Pix3D: Dataset and Methods for Single-Image 3D Shape Modelingcited by 218
  12. 2019Texture Fields: Learning Texture Representations in Function Spacecited by 210
  13. 2018Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Networkcited by 195
  14. 2016Learning Shape Abstractions by Assembling Volumetric Primitivescited by 181
  15. 2019DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracingcited by 179
  16. 2017Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistencycited by 177
  17. 2018Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddingscited by 169
  18. 2019SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimizationcited by 159
  19. 2018Nonlinear 3D Face Morphable Modelcited by 157
  20. 2020Self-supervised Single-view 3D Reconstruction via Semantic Consistencycited by 127
  21. 2017MarrNet: 3D Shape Reconstruction via 2.5D Sketchescited by 108
  22. 2019Pixel2Mesh++: Multi-View 3D Mesh Generation via Deformationcited by 103
  23. 2019Learning to Infer Implicit Surfaces without 3D Supervisioncited by 103
  24. 2020Shape and Viewpoint without Keypointscited by 92
  25. 2019Towards High-fidelity Nonlinear 3D Face Morphable Modelcited by 91
  26. 2019BAE-NET: Branched Autoencoder for Shape Co-Segmentationcited by 89
  27. 2020PatchNets: Patch-Based Generalizable Deep Implicit 3D Shape Representationscited by 81
  28. 2020Articulation-aware Canonical Surface Mappingcited by 80
  29. 2017Neural Face Editing with Intrinsic Image Disentanglingcited by 70
  30. 2018Learning Shape Priors for Single-View 3D Completion and Reconstructioncited by 68
  31. 2018Deep Part Induction from Articulated Object Pairscited by 66
  32. 2019Canonical Surface Mapping via Geometric Cycle Consistencycited by 62
  33. 2018Learning to Reconstruct Shapes from Unseen Classescited by 60
  34. 2014Category-Specific Object Reconstruction from a Single Imagecited by 56
  35. 2018On Learning 3D Face Morphable Model from In-the-wild Imagescited by 54
  36. 2019Combining 3D Morphable Models: A Large scale Face-and-Head Modelcited by 49
  37. 2019Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstructioncited by 28
  38. 2020Learning Implicit Functions for Topology-Varying Dense 3D Shape Correspondencecited by 27

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 5; 2023: 3; 2024: 4; 2025: 2; 2026: 1.

16 papers.

Sort:

By area:

  1. 2022Highly Accurate Dichotomous Image Segmentationcited by 110
  2. 2022Face Relighting with Geometrically Consistent Shadowscited by 43
  3. 2023Learning Clothing and Pose Invariant 3D Shape Representation for Long-Term Person Re-Identificationcited by 34
  4. 2024Multi-view Aggregation Network for Dichotomous Image Segmentationcited by 28
  5. 2022Neural Correspondence Field for Object Pose Estimationcited by 21
  6. 2021Voxel-based 3D Detection and Reconstruction of Multiple Objects from a Single Imagecited by 20
  7. 2024DiffuMatting: Synthesizing Arbitrary Objects with Matting-level Annotationcited by 13
  8. 20222D GANs Meet Unsupervised Single-view 3D Reconstructioncited by 11
  9. 2024High-Precision Dichotomous Image Segmentation via Probing Diffusion Capacitycited by 7
  10. 2023Neural Voting Field for Camera-Space 3D Hand Pose Estimationcited by 6
  11. 2025S3OD: Towards Generalizable Salient Object Detection with Synthetic Datacited by 5
  12. 2024GenUDC: High Quality 3D Mesh Generation with Unsigned Dual Contouring Representationcited by 3
4 more
  1. 2022Learning Implicit Functions for Dense 3D Shape Correspondence of Generic Objectscited by 2
  2. 20233D Semantic Subspace Traverser: Empowering 3D Generative Model with Shape Editing Capabilitycited by 1
  3. 2025LawDIS: Language-Window-based Controllable Dichotomous Image Segmentationcited by 1
  4. 2026Toward Real-World High-Precision Image Matting and Segmentationcited by 0