# Self-supervised Single-view 3D Reconstruction via Semantic Consistency

arXiv: https://arxiv.org/abs/2003.06473 · HTML: https://mlsanity.com/paper/2003.06473/

## Key facts

- Cited by 127 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Mar 2020.

## Built on by (most cited first, 30 of 127)

- [LRM: Large Reconstruction Model for Single Image to 3D](https://mlsanity.com/paper/2311.04400/) (2023)
- [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021)
- [SyncDreamer: Generating Multiview-consistent Images from a Single-view Image](https://mlsanity.com/paper/2309.03453/) (2023)
- [BANMo: Building Animatable 3D Neural Models from Many Casual Videos](https://mlsanity.com/paper/2112.12761/) (2021)
- [LASR: Learning Articulated Shape Reconstruction from a Monocular Video](https://mlsanity.com/paper/2105.02976/) (2021)
- [MagicPony: Learning Articulated 3D Animals in the Wild](https://mlsanity.com/paper/2211.12497/) (2022)
- [HoloDiffusion: Training a 3D Diffusion Model using 2D Images](https://mlsanity.com/paper/2303.16509/) (2023)
- [What's in your hands? 3D Reconstruction of Generic Objects in Hands](https://mlsanity.com/paper/2204.07153/) (2022)
- [SNUG: Self-Supervised Neural Dynamic Garments](https://mlsanity.com/paper/2204.02219/) (2022)
- [A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis](https://mlsanity.com/paper/2110.15678/) (2021)
- [Unsupervised Learning of 3D Object Categories from Videos in the Wild](https://mlsanity.com/paper/2103.16552/) (2021)
- [LASSIE: Learning Articulated Shapes from Sparse Image Ensemble via 3D Part Discovery](https://mlsanity.com/paper/2207.03434/) (2022)
- [Shelf-Supervised Mesh Prediction in the Wild](https://mlsanity.com/paper/2102.06195/) (2021)
- [DOVE: Learning Deformable 3D Objects by Watching Videos](https://mlsanity.com/paper/2107.10844/) (2021)
- [DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision](https://mlsanity.com/paper/2105.06464/) (2021)
- [Category-Level 6D Object Pose Estimation in the Wild: A Semi-Supervised Learning Approach and A New Dataset](https://mlsanity.com/paper/2206.15436/) (2022)
- [Learning the 3D Fauna of the Web](https://mlsanity.com/paper/2401.02400/) (2024)
- [Sketch2Model: View-Aware 3D Modeling from Single Free-Hand Sketches](https://mlsanity.com/paper/2105.06663/) (2021)
- [SparsePose: Sparse-View Camera Pose Regression and Refinement](https://mlsanity.com/paper/2211.16991/) (2022)
- [Generative Occupancy Fields for 3D Surface-Aware Image Synthesis](https://mlsanity.com/paper/2111.00969/) (2021)
- [Reconstructing Animatable Categories from Videos](https://mlsanity.com/paper/2305.06351/) (2023)
- [3D Shape Reconstruction from 2D Images with Disentangled Attribute Flow](https://mlsanity.com/paper/2203.15190/) (2022)
- [REACTO: Reconstructing Articulated Objects from a Single Video](https://mlsanity.com/paper/2404.11151/) (2024)
- [Learning Generative Models of Textured 3D Meshes from Real-World Images](https://mlsanity.com/paper/2103.15627/) (2021)
- [Farm3D: Learning Articulated 3D Animals by Distilling 2D Diffusion](https://mlsanity.com/paper/2304.10535/) (2023)
- [EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild](https://mlsanity.com/paper/2411.14280/) (2024)
- [3D Human Texture Estimation from a Single Image with Transformers](https://mlsanity.com/paper/2109.02563/) (2021)
- [De-rendering 3D Objects in the Wild](https://mlsanity.com/paper/2201.02279/) (2022)
- [Hi-LASSIE: High-Fidelity Articulated Shape and Skeleton Discovery from Sparse Image Ensemble](https://mlsanity.com/paper/2212.11042/) (2022)
- [State of the Art in Dense Monocular Non-Rigid 3D Reconstruction](https://mlsanity.com/paper/2210.15664/) (2022)

## Abstract (arXiv)

We learn a self-supervised, single-view 3D reconstruction model that predicts the 3D mesh shape, texture and camera pose of a target object with a collection of 2D images and silhouettes. The proposed method does not necessitate 3D supervision, manually annotated keypoints, multi-view images of an object or a prior 3D template. The key insight of our work is that objects can be represented as a collection of deformable parts, and each part is semantically coherent across different instances of the same category (e.g., wings on birds and wheels on cars). Therefore, by leveraging self-supervisedly learned part segmentation of a large collection of category-specific images, we can effectively enforce semantic consistency between the reconstructed meshes and the original images. This significantly reduces ambiguities during joint prediction of shape and camera pose of an object, along with texture. To the best of our knowledge, we are the first to try and solve the single-view reconstruction problem without a category-specific template mesh or semantic keypoints. Thus our model can easily generalize to various object categories without such labels, e.g., horses, penguins, etc. Through a variety of experiments on several categories of deformable and rigid objects, we demonstrate that our unsupervised method performs comparably if not better than existing category-specific reconstruction methods learned with supervision.