# An Effective Loss Function for Generating 3D Models from Single 2D Image without Rendering

arXiv: https://arxiv.org/abs/2103.03390 · HTML: https://mlsanity.com/paper/2103.03390/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 39 references that are in the map.
- First posted on arXiv 5 Mar 2021.

## Builds on (most cited first, 27 of 27)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generation](https://mlsanity.com/paper/1802.05384/) (2018)
- [DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction](https://mlsanity.com/paper/1905.10711/) (2019)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer](https://mlsanity.com/paper/1908.01210/) (2019)
- [Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling](https://mlsanity.com/paper/1804.04610/) (2018)
- [Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image](https://mlsanity.com/paper/1905.02722/) (2019)
- [Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs](https://mlsanity.com/paper/1703.09438/) (2017)
- [DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing](https://mlsanity.com/paper/1911.13225/) (2019)
- [Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency](https://mlsanity.com/paper/1704.06254/) (2017)
- [Differentiable Rendering: A Survey](https://mlsanity.com/paper/2006.12057/) (2020)
- [MarrNet: 3D Shape Reconstruction via 2.5D Sketches](https://mlsanity.com/paper/1711.03129/) (2017)
- [Unsupervised Learning of Shape and Pose with Differentiable Point Clouds](https://mlsanity.com/paper/1810.09381/) (2018)
- [Unsupervised Learning of 3D Structure from Images](https://mlsanity.com/paper/1607.00662/) (2016)
- [3DN: 3D Deformation Network](https://mlsanity.com/paper/1903.03322/) (2019)
- [Multi-view Consistency as Supervisory Signal for Learning Shape and Pose Prediction](https://mlsanity.com/paper/1801.03910/) (2018)
- [TextureGAN: Controlling Deep Image Synthesis with Texture Patches](https://mlsanity.com/paper/1706.02823/) (2017)
- [RenderNet: A deep convolutional network for differentiable rendering from 3D shapes](https://mlsanity.com/paper/1806.06575/) (2018)
- [Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction](https://mlsanity.com/paper/1901.05567/) (2019)
- [A survey on Deep Learning Advances on Different 3D Data Representations](https://mlsanity.com/paper/1808.01462/) (2018)
- [IMNet: A Learning Based Detector for Index Modulation Aided MIMO-OFDM Systems](https://mlsanity.com/paper/1911.04133/) (2019)
- [Learning Structral coherence Via Generative Adversarial Network for Single Image Super-Resolution](https://mlsanity.com/paper/2101.10165/) (2021)

## Built on by (most cited first, 7 of 7)

- [State Space Models for Event Cameras](https://mlsanity.com/paper/2402.15584/) (2024)
- [SceneRF: Self-Supervised Monocular 3D Scene Reconstruction with Radiance Fields](https://mlsanity.com/paper/2212.02501/) (2022)
- [ISS: Image as Stepping Stone for Text-Guided 3D Shape Generation](https://mlsanity.com/paper/2209.04145/) (2022)
- [CorticalFlow: A Diffeomorphic Mesh Deformation Module for Cortical Surface Reconstruction](https://mlsanity.com/paper/2206.02374/) (2022)
- [DreamStone: Image as Stepping Stone for Text-Guided 3D Shape Generation](https://mlsanity.com/paper/2303.15181/) (2023)
- [GG-SSMs: Graph-Generating State Space Models](https://mlsanity.com/paper/2412.12423/) (2024)
- [Perturbed State Space Feature Encoders for Optical Flow with Event Cameras](https://mlsanity.com/paper/2504.10669/) (2025)

## Abstract (arXiv)

Differentiable rendering is a very successful technique that applies to a Single-View 3D Reconstruction. Current renderers use losses based on pixels between a rendered image of some 3D reconstructed object and ground-truth images from given matched viewpoints to optimise parameters of the 3D shape. These models require a rendering step, along with visibility handling and evaluation of the shading model. The main goal of this paper is to demonstrate that we can avoid these steps and still get reconstruction results as other state-of-the-art models that are equal or even better than existing category-specific reconstruction methods. First, we use the same CNN architecture for the prediction of a point cloud shape and pose prediction like the one used by Insafutdinov & Dosovitskiy. Secondly, we propose the novel effective loss function that evaluates how well the projections of reconstructed 3D point clouds cover the ground truth object's silhouette. Then we use Poisson Surface Reconstruction to transform the reconstructed point cloud into a 3D mesh. Finally, we perform a GAN-based texture mapping on a particular 3D mesh and produce a textured 3D mesh from a single 2D image. We evaluate our method on different datasets (including ShapeNet, CUB-200-2011, and Pascal3D+) and achieve state-of-the-art results, outperforming all the other supervised and unsupervised methods and 3D representations, all in terms of performance, accuracy, and training time.