# Zero Grads: Learning Local Surrogate Losses for Non-Differentiable Graphics

arXiv: https://arxiv.org/abs/2308.05739 · HTML: https://mlsanity.com/paper/2308.05739/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 107 references that are in the map.
- First posted on arXiv 10 Aug 2023.

## Builds on (most cited first, 30 of 37)

- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Modular Primitives for High-Performance Differentiable Rendering](https://mlsanity.com/paper/2011.03277/) (2020)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [The CMA Evolution Strategy: A Tutorial](https://mlsanity.com/paper/1604.00772/) (2016)
- [Monte Carlo Gradient Estimation in Machine Learning](https://mlsanity.com/paper/1906.10652/) (2019)
- [DiffTaichi: Differentiable Programming for Physical Simulation](https://mlsanity.com/paper/1910.00935/) (2019)
- [Real-time Deep Dynamic Characters](https://mlsanity.com/paper/2105.01794/) (2021)
- [Learning to Control PDEs with Differentiable Physics](https://mlsanity.com/paper/2001.07457/) (2020)
- [A Compositional Object-Based Approach to Learning Physical Dynamics](https://mlsanity.com/paper/1612.00341/) (2016)
- [Flexible Neural Representation for Physics Prediction](https://mlsanity.com/paper/1806.08047/) (2018)
- [Do Differentiable Simulators Give Better Policy Gradients?](https://mlsanity.com/paper/2202.00817/) (2022)
- [Gradients are Not All You Need](https://mlsanity.com/paper/2111.05803/) (2021)
- [On Nesting Monte Carlo Estimators](https://mlsanity.com/paper/1709.06181/) (2017)
- [Deep Parametric Indoor Lighting Estimation](https://mlsanity.com/paper/1910.08812/) (2019)
- [SPNets: Differentiable Fluid Dynamics for Deep Neural Networks](https://mlsanity.com/paper/1806.06094/) (2018)
- [EcoNAS: Finding Proxies for Economical Neural Architecture Search](https://mlsanity.com/paper/2001.01233/) (2020)
- [Deep Shading: Convolutional Neural Networks for Screen-Space Shading](https://mlsanity.com/paper/1603.06078/) (2016)
- [A Versatile Scene Model with Differentiable Visibility Applied to Generative Pose Estimation](https://mlsanity.com/paper/1602.03725/) (2016)
- [PyGAD: An Intuitive Genetic Algorithm Python Library](https://mlsanity.com/paper/2106.06158/) (2021)
- [Bundled Gradients through Contact via Randomized Smoothing](https://mlsanity.com/paper/2109.05143/) (2021)
- [An Inverse Procedural Modeling Pipeline for SVBRDF Maps](https://mlsanity.com/paper/2109.06395/) (2021)
- [PSDR-Room: Single Photo to Scene using Differentiable Rendering](https://mlsanity.com/paper/2307.03244/) (2023)
- [Learning Surrogate Losses](https://mlsanity.com/paper/1905.10108/) (2019)
- [Dr.Jit: A Just-In-Time Compiler for Differentiable Rendering](https://mlsanity.com/paper/2202.01284/) (2022)
- [Autoinverse: Uncertainty Aware Inversion of Neural Networks](https://mlsanity.com/paper/2208.13780/) (2022)
- [DiffTune: Optimizing CPU Simulator Parameters with Learned Differentiable Surrogates](https://mlsanity.com/paper/2010.04017/) (2020)
- [Node Graph Optimization Using Differentiable Proxies](https://mlsanity.com/paper/2207.07684/) (2022)
- [Designing Perceptual Puzzles by Differentiating Probabilistic Programs](https://mlsanity.com/paper/2204.12301/) (2022)

## Built on by (most cited first, 5 of 5)

- [Accurate Differential Operators for Hybrid Neural Fields](https://mlsanity.com/paper/2312.05984/) (2023)
- [Locally Orderless Images for Optimization in Differentiable Rendering](https://mlsanity.com/paper/2503.21931/) (2025)
- [Learning Neural Antiderivatives](https://mlsanity.com/paper/2509.17755/) (2025)
- [Latent Space Reinforcement Learning for Inverse Material Estimation in Food Fracture Simulation](https://mlsanity.com/paper/2606.16870/) (2026)
- [Feature-Guided Diffusion for Non-Differentiable Inverse Rendering](https://mlsanity.com/paper/2607.17411/) (2026)

## Abstract (arXiv)

Gradient-based optimization is now ubiquitous across graphics, but unfortunately can not be applied to problems with undefined or zero gradients. To circumvent this issue, the loss function can be manually replaced by a ``surrogate'' that has similar minima but is differentiable. Our proposed framework, ZeroGrads, automates this process by learning a neural approximation of the objective function, which in turn can be used to differentiate through arbitrary black-box graphics pipelines. We train the surrogate on an actively smoothed version of the objective and encourage locality, focusing the surrogate's capacity on what matters at the current training episode. The fitting is performed online, alongside the parameter optimization, and self-supervised, without pre-computed data or pre-trained models. As sampling the objective is expensive (it requires a full rendering or simulator run), we devise an efficient sampling scheme that allows for tractable run-times and competitive performance at little overhead. We demonstrate optimizing diverse non-convex, non-differentiable black-box problems in graphics, such as visibility in rendering, discrete parameter spaces in procedural modelling or optimal control in physics-driven animation. In contrast to other derivative-free algorithms, our approach scales well to higher dimensions, which we demonstrate on problems with up to 35k interlinked variables.