# Fully Understanding Generic Objects: Modeling, Segmentation, and Reconstruction

arXiv: https://arxiv.org/abs/2104.00858 · HTML: https://mlsanity.com/paper/2104.00858/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 76 references that are in the map.
- First posted on arXiv 2 Apr 2021.

## Builds on (most cited first, 30 of 57)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [Convolutional Occupancy Networks](https://mlsanity.com/paper/2003.04618/) (2020)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [U2-Net: Going Deeper with Nested U-Structure for Salient Object Detection](https://mlsanity.com/paper/2005.09007/) (2020)
- [Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion](https://mlsanity.com/paper/2003.01456/) (2020)
- [DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction](https://mlsanity.com/paper/1905.10711/) (2019)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [What Do Single-view 3D Reconstruction Networks Learn?](https://mlsanity.com/paper/1905.03678/) (2019)
- [M3D-RPN: Monocular 3D Region Proposal Network for Object Detection](https://mlsanity.com/paper/1907.06038/) (2019)
- [Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling](https://mlsanity.com/paper/1804.04610/) (2018)
- [Texture Fields: Learning Texture Representations in Function Space](https://mlsanity.com/paper/1905.07259/) (2019)
- [Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network](https://mlsanity.com/paper/1803.07835/) (2018)
- [Learning Shape Abstractions by Assembling Volumetric Primitives](https://mlsanity.com/paper/1612.00404/) (2016)
- [DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing](https://mlsanity.com/paper/1911.13225/) (2019)
- [Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency](https://mlsanity.com/paper/1704.06254/) (2017)
- [Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings](https://mlsanity.com/paper/1803.08495/) (2018)
- [SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimization](https://mlsanity.com/paper/1912.07109/) (2019)

## Built on by (most cited first, 16 of 16)

- [Highly Accurate Dichotomous Image Segmentation](https://mlsanity.com/paper/2203.03041/) (2022)
- [Face Relighting with Geometrically Consistent Shadows](https://mlsanity.com/paper/2203.16681/) (2022)
- [Learning Clothing and Pose Invariant 3D Shape Representation for Long-Term Person Re-Identification](https://mlsanity.com/paper/2308.10658/) (2023)
- [Multi-view Aggregation Network for Dichotomous Image Segmentation](https://mlsanity.com/paper/2404.07445/) (2024)
- [Neural Correspondence Field for Object Pose Estimation](https://mlsanity.com/paper/2208.00113/) (2022)
- [Voxel-based 3D Detection and Reconstruction of Multiple Objects from a Single Image](https://mlsanity.com/paper/2111.03098/) (2021)
- [DiffuMatting: Synthesizing Arbitrary Objects with Matting-level Annotation](https://mlsanity.com/paper/2403.06168/) (2024)
- [2D GANs Meet Unsupervised Single-view 3D Reconstruction](https://mlsanity.com/paper/2207.10183/) (2022)
- [High-Precision Dichotomous Image Segmentation via Probing Diffusion Capacity](https://mlsanity.com/paper/2410.10105/) (2024)
- [Neural Voting Field for Camera-Space 3D Hand Pose Estimation](https://mlsanity.com/paper/2305.04328/) (2023)
- [S3OD: Towards Generalizable Salient Object Detection with Synthetic Data](https://mlsanity.com/paper/2510.21605/) (2025)
- [GenUDC: High Quality 3D Mesh Generation with Unsigned Dual Contouring Representation](https://mlsanity.com/paper/2410.17802/) (2024)
- [Learning Implicit Functions for Dense 3D Shape Correspondence of Generic Objects](https://mlsanity.com/paper/2212.14276/) (2022)
- [3D Semantic Subspace Traverser: Empowering 3D Generative Model with Shape Editing Capability](https://mlsanity.com/paper/2307.14051/) (2023)
- [LawDIS: Language-Window-based Controllable Dichotomous Image Segmentation](https://mlsanity.com/paper/2508.01152/) (2025)
- [Toward Real-World High-Precision Image Matting and Segmentation](https://mlsanity.com/paper/2601.12080/) (2026)

## Abstract (arXiv)

Inferring 3D structure of a generic object from a 2D image is a long-standing objective of computer vision. Conventional approaches either learn completely from CAD-generated synthetic data, which have difficulty in inference from real images, or generate 2.5D depth image via intrinsic decomposition, which is limited compared to the full 3D reconstruction. One fundamental challenge lies in how to leverage numerous real 2D images without any 3D ground truth. To address this issue, we take an alternative approach with semi-supervised learning. That is, for a 2D image of a generic object, we decompose it into latent representations of category, shape and albedo, lighting and camera projection matrix, decode the representations to segmented 3D shape and albedo respectively, and fuse these components to render an image well approximating the input image. Using a category-adaptive 3D joint occupancy field (JOF), we show that the complete shape and albedo modeling enables us to leverage real 2D images in both modeling and model fitting. The effectiveness of our approach is demonstrated through superior 3D reconstruction from a single image, being either synthetic or real, and shape segmentation.