# Two-stage Synthetic Supervising and Multi-view Consistency Self-supervising based Animal 3D Reconstruction by Single Image

arXiv: https://arxiv.org/abs/2311.13199 · HTML: https://mlsanity.com/paper/2311.13199/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 38 references that are in the map.
- First posted on arXiv 22 Nov 2023.

## Builds on (most cited first, 30 of 32)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling](https://mlsanity.com/paper/1610.07584/) (2016)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer](https://mlsanity.com/paper/1908.01210/) (2019)
- [Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency](https://mlsanity.com/paper/1704.06254/) (2017)
- [Learning a Predictable and Generative Vector Representation for Objects](https://mlsanity.com/paper/1603.08637/) (2016)
- [Self-supervised Single-view 3D Reconstruction via Semantic Consistency](https://mlsanity.com/paper/2003.06473/) (2020)
- [Visual Object Networks: Image Generation with Disentangled 3D Representation](https://mlsanity.com/paper/1812.02725/) (2018)
- [Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision](https://mlsanity.com/paper/1612.00814/) (2016)
- [Deep Mesh Reconstruction from Single RGB Images via Topology Modification Networks](https://mlsanity.com/paper/1909.00321/) (2019)
- [Pixel2Mesh++: Multi-View 3D Mesh Generation via Deformation](https://mlsanity.com/paper/1908.01491/) (2019)
- [Learning to Infer Implicit Surfaces without 3D Supervision](https://mlsanity.com/paper/1911.00767/) (2019)
- [Hierarchical Surface Prediction for 3D Object Reconstruction](https://mlsanity.com/paper/1704.00710/) (2017)
- [Learning single-image 3D reconstruction by generative modelling of shape, pose and shading](https://mlsanity.com/paper/1901.06447/) (2019)
- [Learning View Priors for Single-view 3D Reconstruction](https://mlsanity.com/paper/1811.10719/) (2018)
- [SilNet : Single- and Multi-View Reconstruction by Learning from Silhouettes](https://mlsanity.com/paper/1711.07888/) (2017)
- [Rethinking Reprojection: Closing the Loop for Pose-aware ShapeReconstruction from a Single Image](https://mlsanity.com/paper/1707.04682/) (2017)
- [Weakly supervised 3D Reconstruction with Adversarial Constraint](https://mlsanity.com/paper/1705.10904/) (2017)
- [Learning to Generate and Reconstruct 3D Meshes with only 2D Supervision](https://mlsanity.com/paper/1807.09259/) (2018)
- [Product Re-identification System in Fully Automated Defect Detection](https://mlsanity.com/paper/2112.10324/) (2021)

## Abstract (arXiv)

Pixel-aligned Implicit Function (PIFu) effectively captures subtle variations in body shape within a low-dimensional space through extensive training with human 3D scans, its application to live animals presents formidable challenges due to the difficulty of obtaining animal cooperation for 3D scanning. To address this challenge, we propose the combination of two-stage supervised and self-supervised training to address the challenge of obtaining animal cooperation for 3D scanning. In the first stage, we leverage synthetic animal models for supervised learning. This allows the model to learn from a diverse set of virtual animal instances. In the second stage, we use 2D multi-view consistency as a self-supervised training method. This further enhances the model's ability to reconstruct accurate and realistic 3D shape and texture from largely available single-view images of real animals. The results of our study demonstrate that our approach outperforms state-of-the-art methods in both quantitative and qualitative aspects of bird 3D digitization. The source code is available at https://github.com/kuangzijian/drifu-for-animals.