# Pix2Vex: Image-to-Geometry Reconstruction using a Smooth Differentiable Renderer

arXiv: https://arxiv.org/abs/1903.11149 · HTML: https://mlsanity.com/paper/1903.11149/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Mar 2019.

## Built on by (most cited first, 18 of 18)

- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [Mixture of Volumetric Primitives for Efficient Neural Rendering](https://mlsanity.com/paper/2103.01954/) (2021)
- [GES: Generalized Exponential Splatting for Efficient Radiance Field Rendering](https://mlsanity.com/paper/2402.10128/) (2024)
- [3D Convex Splatting: Radiance Field Rendering with 3D Smooth Convexes](https://mlsanity.com/paper/2411.14974/) (2024)
- [Learning with Algorithmic Supervision via Continuous Relaxations](https://mlsanity.com/paper/2110.05651/) (2021)
- [PAPR: Proximity Attention Point Rendering](https://mlsanity.com/paper/2307.11086/) (2023)
- [GenDR: A Generalized Differentiable Renderer](https://mlsanity.com/paper/2204.13845/) (2022)
- [Plateau-reduced Differentiable Path Tracing](https://mlsanity.com/paper/2211.17263/) (2022)
- [Inverse Global Illumination using a Neural Radiometric Prior](https://mlsanity.com/paper/2305.02192/) (2023)
- [Generalizing Stochastic Smoothing for Differentiation and Gradient Estimation](https://mlsanity.com/paper/2410.08125/) (2024)
- [One Ring to Rule Them All: a simple solution to multi-view 3D-Reconstruction of shapes with unknown BRDF via a small Recurrent ResNet](https://mlsanity.com/paper/2104.05014/) (2021)
- [Newton Losses: Using Curvature Information for Learning with Differentiable Algorithms](https://mlsanity.com/paper/2410.19055/) (2024)
- [Differentiable Neural Radiosity](https://mlsanity.com/paper/2201.13190/) (2022)
- [Differentiable Rendering for Synthetic Aperture Radar Imagery](https://mlsanity.com/paper/2204.01248/) (2022)
- [Learning to Rasterize Differentiably](https://mlsanity.com/paper/2211.13333/) (2022)
- [Towards Real-Time Generation of Delay-Compensated Video Feeds for Outdoor Mobile Robot Teleoperation](https://mlsanity.com/paper/2409.09921/) (2024)
- [Gaussian Mesh Renderer for Lightweight Differentiable Rendering](https://mlsanity.com/paper/2602.14493/) (2026)
- [DiffBMP: Differentiable Rendering with Bitmap Primitives](https://mlsanity.com/paper/2602.22625/) (2026)

## Abstract (arXiv)

The long-coveted task of reconstructing 3D geometry from images is still a standing problem. In this paper, we build on the power of neural networks and introduce Pix2Vex, a network trained to convert camera-captured images into 3D geometry. We present a novel differentiable renderer (DR) as a forward validation means during training. Our key insight is that DRs produce images of a particular appearance, different from typical input images. Hence, we propose adding an image-to-image translation component, converting between these rendering styles. This translation closes the training loop, while allowing to use minimal supervision only, without needing any 3D model as ground truth. Unlike state-of-the-art methods, our DR is C^∞ smooth and thus does not display any discontinuities at occlusions or dis-occlusions. Through our novel training scheme, our network can train on different types of images, where previous work can typically only train on images of a similar appearance to those rendered by a DR.