# Differentiable Stereopsis: Meshes from multiple views using differentiable rendering

arXiv: https://arxiv.org/abs/2110.05472 · HTML: https://mlsanity.com/paper/2110.05472/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 62 references that are in the map.
- First posted on arXiv 11 Oct 2021.

## Builds on (most cited first, 22 of 22)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [MegaDepth: Learning Single-View Depth Prediction from Internet Photos](https://mlsanity.com/paper/1804.00607/) (2018)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose](https://mlsanity.com/paper/1803.02276/) (2018)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer](https://mlsanity.com/paper/1908.01210/) (2019)
- [Self-supervised Single-view 3D Reconstruction via Semantic Consistency](https://mlsanity.com/paper/2003.06473/) (2020)
- [3D Shape Induction from 2D Views of Multiple Objects](https://mlsanity.com/paper/1612.05872/) (2016)
- [Shape and Viewpoint without Keypoints](https://mlsanity.com/paper/2007.10982/) (2020)
- [Articulation-aware Canonical Surface Mapping](https://mlsanity.com/paper/2004.00614/) (2020)
- [Dense Hybrid Recurrent Multi-view Stereo Net with Dynamic Consistency Checking](https://mlsanity.com/paper/2007.10872/) (2020)
- [Shelf-Supervised Mesh Prediction in the Wild](https://mlsanity.com/paper/2102.06195/) (2021)
- [Category-Specific Object Reconstruction from a Single Image](https://mlsanity.com/paper/1411.6069/) (2014)
- [Implicit Mesh Reconstruction from Unannotated Image Collections](https://mlsanity.com/paper/2007.08504/) (2020)

## Built on by (most cited first, 26 of 26)

- [SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstruction](https://mlsanity.com/paper/2212.00792/) (2022)
- [Unique3D: High-Quality and Efficient 3D Mesh Generation from a Single Image](https://mlsanity.com/paper/2405.20343/) (2024)
- [Cameras as Rays: Pose Estimation via Ray Diffusion](https://mlsanity.com/paper/2402.14817/) (2024)
- [MagicPony: Learning Articulated 3D Animals in the Wild](https://mlsanity.com/paper/2211.12497/) (2022)
- [RelPose: Predicting Probabilistic Relative Rotation for Single Objects in the Wild](https://mlsanity.com/paper/2208.05963/) (2022)
- [Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materials](https://mlsanity.com/paper/2407.02445/) (2024)
- [Differentiable Blocks World: Qualitative 3D Decomposition by Rendering Primitives](https://mlsanity.com/paper/2307.05473/) (2023)
- [Looking Through the Glass: Neural Surface Reconstruction Against High Specular Reflections](https://mlsanity.com/paper/2304.08706/) (2023)
- [SPARF: Large-Scale Learning of 3D Sparse Radiance Fields from Few Input Images](https://mlsanity.com/paper/2212.09100/) (2022)
- [HollowNeRF: Pruning Hashgrid-Based NeRFs with Trainable Collision Mitigation](https://mlsanity.com/paper/2308.10122/) (2023)
- [Geometry-biased Transformers for Novel View Synthesis](https://mlsanity.com/paper/2301.04650/) (2023)
- [GLS: Geometry-aware 3D Language Gaussian Splatting](https://mlsanity.com/paper/2411.18066/) (2024)
- [TMO: Textured Mesh Acquisition of Objects with a Mobile Device by using Differentiable Rendering](https://mlsanity.com/paper/2303.15060/) (2023)
- [Sur2f: A Hybrid Representation for High-Quality and Efficient Surface Reconstruction from Multi-view Images](https://mlsanity.com/paper/2401.03704/) (2024)
- [The More You See in 2D, the More You Perceive in 3D](https://mlsanity.com/paper/2404.03652/) (2024)
- [Sparse 3D Reconstruction via Object-Centric Ray Sampling](https://mlsanity.com/paper/2309.03008/) (2023)
- [A Review of Deep Learning-Powered Mesh Reconstruction Methods](https://mlsanity.com/paper/2303.02879/) (2023)
- [GeoScaler: Geometry and Rendering-Aware Downsampling of 3D Mesh Textures](https://mlsanity.com/paper/2311.16581/) (2023)
- [Meta 3D Gen](https://mlsanity.com/paper/2407.02599/) (2024)
- [IM360: Large-scale Indoor Mapping with 360 Cameras](https://mlsanity.com/paper/2502.12545/) (2025)
- [3D Surface Reconstruction in the Wild by Deforming Shape Priors from Synthetic Data](https://mlsanity.com/paper/2302.12883/) (2023)
- [FastMESH: Fast Surface Reconstruction by Hexagonal Mesh-based Neural Rendering](https://mlsanity.com/paper/2305.17858/) (2023)
- [Partial-View Object View Synthesis via Filtered Inversion](https://mlsanity.com/paper/2304.00673/) (2023)
- [Disentangled Geometry and Appearance for Efficient Multi-View Surface Reconstruction and Rendering](https://mlsanity.com/paper/2508.17436/) (2025)
- [ProFusion: 3D Reconstruction of Protein Complex Structures from Multi-view AFM Images](https://mlsanity.com/paper/2509.15242/) (2025)
- [Category-Level 3D Correspondence in Camera Space via Morphable Object Priors](https://mlsanity.com/paper/2605.28257/) (2026)

## Abstract (arXiv)

We propose Differentiable Stereopsis, a multi-view stereo approach that reconstructs shape and texture from few input views and noisy cameras. We pair traditional stereopsis and modern differentiable rendering to build an end-to-end model which predicts textured 3D meshes of objects with varying topologies and shape. We frame stereopsis as an optimization problem and simultaneously update shape and cameras via simple gradient descent. We run an extensive quantitative analysis and compare to traditional multi-view stereo techniques and state-of-the-art learning based methods. We show compelling reconstructions on challenging real-world scenes and for an abundance of object types with complex shape, topology and texture. Project webpage: https://shubham-goel.github.io/ds/