# Multi-View Mesh Reconstruction with Neural Deferred Shading

arXiv: https://arxiv.org/abs/2212.04386 · HTML: https://mlsanity.com/paper/2212.04386/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 79 references that are in the map.
- First posted on arXiv 8 Dec 2022.

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Convolutional Occupancy Networks](https://mlsanity.com/paper/2003.04618/) (2020)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction](https://mlsanity.com/paper/2104.10078/) (2021)
- [Modular Primitives for High-Performance Differentiable Rendering](https://mlsanity.com/paper/2011.03277/) (2020)
- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [Deferred Neural Rendering: Image Synthesis using Neural Textures](https://mlsanity.com/paper/1904.12356/) (2019)
- [Neural Lumigraph Rendering](https://mlsanity.com/paper/2103.11571/) (2021)
- [Pixel2Mesh++: Multi-View 3D Mesh Generation via Deformation](https://mlsanity.com/paper/1908.01491/) (2019)
- [Learning to Infer Implicit Surfaces without 3D Supervision](https://mlsanity.com/paper/1911.00767/) (2019)
- [A Survey on Deep Learning Techniques for Stereo-based Depth Estimation](https://mlsanity.com/paper/2006.02535/) (2020)
- [Image-based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era](https://mlsanity.com/paper/1906.06543/) (2019)
- [Unified Shape and SVBRDF Recovery using Differentiable Monte Carlo Rendering](https://mlsanity.com/paper/2103.15208/) (2021)
- [Neural Subdivision](https://mlsanity.com/paper/2005.01819/) (2020)
- [Neural Volume Rendering: NeRF And Beyond](https://mlsanity.com/paper/2101.05204/) (2020)
- [Photometric Mesh Optimization for Video-Aligned 3D Object Reconstruction](https://mlsanity.com/paper/1903.08642/) (2019)
- [Appearance-Driven Automatic 3D Model Simplification](https://mlsanity.com/paper/2104.03989/) (2021)
- [Differentiable Refraction-Tracing for Mesh Reconstruction of Transparent Objects](https://mlsanity.com/paper/2009.09144/) (2020)

## Built on by (most cited first, 30 of 44)

- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers](https://mlsanity.com/paper/2312.09147/) (2023)
- [SAM 3D: 3Dfy Anything in Images](https://mlsanity.com/paper/2511.16624/) (2025)
- [Unique3D: High-Quality and Efficient 3D Mesh Generation from a Single Image](https://mlsanity.com/paper/2405.20343/) (2024)
- [Multiview Compressive Coding for 3D Reconstruction](https://mlsanity.com/paper/2301.08247/) (2023)
- [FLARE: Fast Learning of Animatable and Relightable Mesh Avatars](https://mlsanity.com/paper/2310.17519/) (2023)
- [Reconstructing Hand-Held Objects in 3D from Images and Videos](https://mlsanity.com/paper/2404.06507/) (2024)
- [TetSphere Splatting: Representing High-Quality Geometry with Lagrangian Volumetric Meshes](https://mlsanity.com/paper/2405.20283/) (2024)
- [Chupa: Carving 3D Clothed Humans from Skinned Shape Priors using 2D Diffusion Probabilistic Models](https://mlsanity.com/paper/2305.11870/) (2023)
- [Looking Through the Glass: Neural Surface Reconstruction Against High Specular Reflections](https://mlsanity.com/paper/2304.08706/) (2023)
- [GarmentDreamer: 3DGS Guided Garment Synthesis with Diverse Geometry and Texture Details](https://mlsanity.com/paper/2405.12420/) (2024)
- [Tree-D Fusion: Simulation-Ready Tree Dataset from Single Images with Diffusion Priors](https://mlsanity.com/paper/2407.10330/) (2024)
- [Beyond Pixel Histories: World Models with Persistent 3D State](https://mlsanity.com/paper/2603.03482/) (2026)
- [CanonicalFusion: Generating Drivable 3D Human Avatars from Multiple Images](https://mlsanity.com/paper/2407.04345/) (2024)
- [HeadEvolver: Text to Head Avatars via Expressive and Attribute-Preserving Mesh Deformation](https://mlsanity.com/paper/2403.09326/) (2024)
- [Explicit Neural Surfaces: Learning Continuous Geometry With Deformation Fields](https://mlsanity.com/paper/2306.02956/) (2023)
- [MVHuman: Tailoring 2D Diffusion with Multi-view Sampling For Realistic 3D Human Generation](https://mlsanity.com/paper/2312.10120/) (2023)
- [High-Quality Mesh Blendshape Generation from Face Videos via Neural Inverse Rendering](https://mlsanity.com/paper/2401.08398/) (2024)
- [Sur2f: A Hybrid Representation for High-Quality and Efficient Surface Reconstruction from Multi-view Images](https://mlsanity.com/paper/2401.03704/) (2024)
- [Differentiable Shadow Mapping for Efficient Inverse Graphics](https://mlsanity.com/paper/2308.10896/) (2023)
- [Sparse 3D Reconstruction via Object-Centric Ray Sampling](https://mlsanity.com/paper/2309.03008/) (2023)
- [VaseVQA-3D: Benchmarking 3D VLMs on Ancient Greek Pottery](https://mlsanity.com/paper/2510.04479/) (2025)
- [DiffSoup: Direct Differentiable Rasterization of Triangle Soup for Extreme Radiance Field Simplification](https://mlsanity.com/paper/2603.27151/) (2026)
- [A Review of Deep Learning-Powered Mesh Reconstruction Methods](https://mlsanity.com/paper/2303.02879/) (2023)
- [Fast-SAM3D: 3Dfy Anything in Images but Faster](https://mlsanity.com/paper/2602.05293/) (2026)
- [Optimization-Based Eye Tracking using Deflectometric Information](https://mlsanity.com/paper/2303.04997/) (2023)
- [FastMESH: Fast Surface Reconstruction by Hexagonal Mesh-based Neural Rendering](https://mlsanity.com/paper/2305.17858/) (2023)
- [REFRAME: Reflective Surface Real-Time Rendering for Mobile Devices](https://mlsanity.com/paper/2403.16481/) (2024)
- [DNRSelect: Active Best View Selection for Deferred Neural Rendering](https://mlsanity.com/paper/2501.12150/) (2025)
- [A Text-to-3D Framework for Joint Generation of CG-Ready Humans and Compatible Garments](https://mlsanity.com/paper/2503.12052/) (2025)

## Abstract (arXiv)

We propose an analysis-by-synthesis method for fast multi-view 3D reconstruction of opaque objects with arbitrary materials and illumination. State-of-the-art methods use both neural surface representations and neural rendering. While flexible, neural surface representations are a significant bottleneck in optimization runtime. Instead, we represent surfaces as triangle meshes and build a differentiable rendering pipeline around triangle rasterization and neural shading. The renderer is used in a gradient descent optimization where both a triangle mesh and a neural shader are jointly optimized to reproduce the multi-view images. We evaluate our method on a public 3D reconstruction dataset and show that it can match the reconstruction accuracy of traditional baselines and neural approaches while surpassing them in optimization runtime. Additionally, we investigate the shader and find that it learns an interpretable representation of appearance, enabling applications such as 3D material editing.