# Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction

arXiv: https://arxiv.org/abs/1901.05567 · HTML: https://mlsanity.com/paper/1901.05567/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jan 2019.

## Built on by (most cited first, 28 of 28)

- [PixelSynth: Generating a 3D-Consistent Experience from a Single Image](https://mlsanity.com/paper/2108.05892/) (2021)
- [Multiview Compressive Coding for 3D Reconstruction](https://mlsanity.com/paper/2301.08247/) (2023)
- [FWD: Real-time Novel View Synthesis with Forward Warping and Depth](https://mlsanity.com/paper/2206.08355/) (2022)
- [Unsupervised Discovery and Composition of Object Light Fields](https://mlsanity.com/paper/2205.03923/) (2022)
- [Diffusion Time-step Curriculum for One Image to 3D Generation](https://mlsanity.com/paper/2404.04562/) (2024)
- [Learning 3D Object Shape and Layout without 3D Supervision](https://mlsanity.com/paper/2206.07028/) (2022)
- [Inverting Generative Adversarial Renderer for Face Reconstruction](https://mlsanity.com/paper/2105.02431/) (2021)
- [EasyHeC: Accurate and Automatic Hand-eye Calibration via Differentiable Rendering and Space Exploration](https://mlsanity.com/paper/2305.01191/) (2023)
- [Adversarial Attacks on Camera-LiDAR Models for 3D Car Detection](https://mlsanity.com/paper/2103.09448/) (2021)
- [Fully Understanding Generic Objects: Modeling, Segmentation, and Reconstruction](https://mlsanity.com/paper/2104.00858/) (2021)
- [Towards Universal Physical Attacks On Cascaded Camera-Lidar 3D Object Detection Models](https://mlsanity.com/paper/2101.10747/) (2021)
- [GlobalMapNet: An Online Framework for Vectorized Global HD Map Construction](https://mlsanity.com/paper/2409.10063/) (2024)
- [HiFECap: Monocular High-Fidelity and Expressive Capture of Human Performances](https://mlsanity.com/paper/2210.05665/) (2022)
- [Deep Feedback Inverse Problem Solver](https://mlsanity.com/paper/2101.07719/) (2021)
- [2D GANs Meet Unsupervised Single-view 3D Reconstruction](https://mlsanity.com/paper/2207.10183/) (2022)
- [Inverse Global Illumination using a Neural Radiometric Prior](https://mlsanity.com/paper/2305.02192/) (2023)
- [Monocular 3D Object Reconstruction with GAN Inversion](https://mlsanity.com/paper/2207.10061/) (2022)
- [An Effective Loss Function for Generating 3D Models from Single 2D Image without Rendering](https://mlsanity.com/paper/2103.03390/) (2021)
- [Diffusion Posterior Illumination for Ambiguity-aware Inverse Rendering](https://mlsanity.com/paper/2310.00362/) (2023)
- [Recognizing Scenes from Novel Viewpoints](https://mlsanity.com/paper/2112.01520/) (2021)
- [Towards 3D Object Detection with 2D Supervision](https://mlsanity.com/paper/2211.08287/) (2022)
- [Towards Scalable Multi-View Reconstruction of Geometry and Materials](https://mlsanity.com/paper/2306.03747/) (2023)
- [Differentiable Neural Radiosity](https://mlsanity.com/paper/2201.13190/) (2022)
- [IPoD: Implicit Field Learning with Point Diffusion for Generalizable 3D Object Reconstruction from Single RGB-D Images](https://mlsanity.com/paper/2404.00269/) (2024)
- [Physically Accurate Differentiable Inverse Rendering for Radio Frequency Digital Twin](https://mlsanity.com/paper/2603.18026/) (2026)
- [Using Adaptive Gradient for Texture Learning in Single-View 3D Reconstruction](https://mlsanity.com/paper/2104.14169/) (2021)
- [Neural Implicit 3D Shapes from Single Images with Spatial Patterns](https://mlsanity.com/paper/2106.03087/) (2021)
- [Scene-SAM3D: Multi-View Scene Asset Generation Without Fine-Tuning](https://mlsanity.com/paper/2607.16805/) (2026)

## Abstract (arXiv)

Rendering is the process of generating 2D images from 3D assets, simulated in a virtual environment, typically with a graphics pipeline. By inverting such renderer, one can think of a learning approach to predict a 3D shape from an input image. However, standard rendering pipelines involve a fundamental discretization step called rasterization, which prevents the rendering process to be differentiable, hence suitable for learning. We present the first non-parametric and truly differentiable rasterizer based on silhouettes. Our method enables unsupervised learning for high-quality 3D mesh reconstruction from a single image. We call our framework `soft rasterizer' as it provides an accurate soft approximation of the standard rasterizer. The key idea is to fuse the probabilistic contributions of all mesh triangles with respect to the rendered pixels. When combined with a mesh generator in a deep neural network, our soft rasterizer is able to generate an approximated silhouette of the generated polygon mesh in the forward pass. The rendering loss is back-propagated to supervise the mesh generation without the need of 3D training data. Experimental results demonstrate that our approach significantly outperforms the state-of-the-art unsupervised techniques, both quantitatively and qualitatively. We also show that our soft rasterizer can achieve comparable results to the cutting-edge supervised learning method and in various cases even better ones, especially for real-world data.