# Multiview Textured Mesh Recovery by Differentiable Rendering

arXiv: https://arxiv.org/abs/2205.12468 · HTML: https://mlsanity.com/paper/2205.12468/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 48 references that are in the map.
- First posted on arXiv 25 May 2022.

## Builds on (most cited first, 19 of 19)

- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction](https://mlsanity.com/paper/2104.10078/) (2021)
- [Modular Primitives for High-Performance Differentiable Rendering](https://mlsanity.com/paper/2011.03277/) (2020)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting](https://mlsanity.com/paper/2104.00674/) (2021)
- [DeepMVS: Learning Multi-view Stereopsis](https://mlsanity.com/paper/1804.00650/) (2018)
- [Recurrent MVSNet for High-resolution Multi-view Stereo Depth Inference](https://mlsanity.com/paper/1902.10556/) (2019)
- [Differentiable Surface Splatting for Point-based Geometry Processing](https://mlsanity.com/paper/1906.04173/) (2019)
- [Shape As Points: A Differentiable Poisson Solver](https://mlsanity.com/paper/2106.03452/) (2021)
- [DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing](https://mlsanity.com/paper/1911.13225/) (2019)
- [SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimization](https://mlsanity.com/paper/1912.07109/) (2019)
- [MeshSDF: Differentiable Iso-Surface Extraction](https://mlsanity.com/paper/2006.03997/) (2020)
- [Visibility-aware Multi-view Stereo Network](https://mlsanity.com/paper/2008.07928/) (2020)
- [Learning Signed Distance Field for Multi-view Surface Reconstruction](https://mlsanity.com/paper/2108.09964/) (2021)
- [Unified Shape and SVBRDF Recovery using Differentiable Monte Carlo Rendering](https://mlsanity.com/paper/2103.15208/) (2021)
- [SPU-Net: Self-Supervised Point Cloud Upsampling by Coarse-to-Fine Reconstruction with Self-Projection Optimization](https://mlsanity.com/paper/2012.04439/) (2020)

## Built on by (most cited first, 8 of 8)

- [StreetSurfGS: Scalable Urban Street Surface Reconstruction with Planar-based Gaussian Splatting](https://mlsanity.com/paper/2410.04354/) (2024)
- [TMO: Textured Mesh Acquisition of Objects with a Mobile Device by using Differentiable Rendering](https://mlsanity.com/paper/2303.15060/) (2023)
- [NeuS-PIR: Learning Relightable Neural Surface using Pre-Integrated Rendering](https://mlsanity.com/paper/2306.07632/) (2023)
- [Dynamic Appearance Particle Neural Radiance Field](https://mlsanity.com/paper/2310.07916/) (2023)
- [Point2Quad: Generating Quad Meshes from Point Clouds via Face Prediction](https://mlsanity.com/paper/2504.19545/) (2025)
- [Fine-Grained Multi-View Hand Reconstruction Using Inverse Rendering](https://mlsanity.com/paper/2407.05680/) (2024)
- [GBR: Generative Bundle Refinement for High-fidelity Gaussian Splatting with Enhanced Mesh Reconstruction](https://mlsanity.com/paper/2412.05908/) (2024)
- [OrthoGAN:High-Precision Image Generation for Teeth Orthodontic Visualization](https://mlsanity.com/paper/2212.14162/) (2022)

## Abstract (arXiv)

Although having achieved the promising results on shape and color recovery through self-supervision, the multi-layer perceptrons-based methods usually suffer from heavy computational cost on learning the deep implicit surface representation. Since rendering each pixel requires a forward network inference, it is very computational intensive to synthesize a whole image. To tackle these challenges, we propose an effective coarse-to-fine approach to recover the textured mesh from multi-views in this paper. Specifically, a differentiable Poisson Solver is employed to represent the object's shape, which is able to produce topology-agnostic and watertight surfaces. To account for depth information, we optimize the shape geometry by minimizing the differences between the rendered mesh and the predicted depth from multi-view stereo. In contrast to the implicit neural representation on shape and color, we introduce a physically based inverse rendering scheme to jointly estimate the environment lighting and object's reflectance, which is able to render the high resolution image at real-time. The texture of the reconstructed mesh is interpolated from a learnable dense texture grid. We have conducted the extensive experiments on several multi-view stereo datasets, whose promising results demonstrate the efficacy of our proposed approach. The code is available at https://github.com/l1346792580123/diff.