# Transforming a Non-Differentiable Rasterizer into a Differentiable One with Stochastic Gradient Estimation

arXiv: https://arxiv.org/abs/2404.09758 · HTML: https://mlsanity.com/paper/2404.09758/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Apr 2024.

## Built on by (most cited first, 5 of 5)

- [StochasticSplats: Stochastic Rasterization for Sorting-Free 3D Gaussian Splatting](https://mlsanity.com/paper/2503.24366/) (2025)
- [DiffCSG: Differentiable CSG via Rasterization](https://mlsanity.com/paper/2409.01421/) (2024)
- [2D Neural Fields with Learned Discontinuities](https://mlsanity.com/paper/2408.00771/) (2024)
- [Locally Orderless Images for Optimization in Differentiable Rendering](https://mlsanity.com/paper/2503.21931/) (2025)
- [Meshtryoshka: Differentiable Rendering of Real-World Scenes via Mesh Rasterization](https://mlsanity.com/paper/2606.28622/) (2026)

## Abstract (arXiv)

We show how to transform a non-differentiable rasterizer into a differentiable one with minimal engineering efforts and no external dependencies (no Pytorch/Tensorflow). We rely on Stochastic Gradient Estimation, a technique that consists of rasterizing after randomly perturbing the scene's parameters such that their gradient can be stochastically estimated and descended. This method is simple and robust but does not scale in dimensionality (number of scene parameters). Our insight is that the number of parameters contributing to a given rasterized pixel is bounded. Estimating and averaging gradients on a per-pixel basis hence bounds the dimensionality of the underlying optimization problem and makes the method scalable. Furthermore, it is simple to track per-pixel contributing parameters by rasterizing ID- and UV-buffers, which are trivial additions to a rasterization engine if not already available. With these minor modifications, we obtain an in-engine optimizer for 3D assets with millions of geometry and texture parameters.