# Instead of Rewriting Foreign Code for Machine Learning, Automatically Synthesize Fast Gradients

arXiv: https://arxiv.org/abs/2010.01709 · HTML: https://mlsanity.com/paper/2010.01709/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Oct 2020.

## Built on by (most cited first, 27 of 27)

- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [Differentiable Programming for Earth System Modeling](https://mlsanity.com/paper/2208.13825/) (2022)
- [New directions for surrogate models and differentiable programming for High Energy Physics detector simulation](https://mlsanity.com/paper/2203.08806/) (2022)
- [A Review of Change of Variable Formulas for Generative Modeling](https://mlsanity.com/paper/2308.02652/) (2023)
- [Closed-Form Diffeomorphic Transformations for Time Series Alignment](https://mlsanity.com/paper/2206.08107/) (2022)
- [Swift for TensorFlow: A portable, flexible platform for deep learning](https://mlsanity.com/paper/2102.13243/) (2021)
- [Magellan: Autonomous Discovery of Novel Compiler Optimization Heuristics with AlphaEvolve](https://mlsanity.com/paper/2601.21096/) (2026)
- [Sparser, Better, Faster, Stronger: Sparsity Detection for Efficient Automatic Differentiation](https://mlsanity.com/paper/2501.17737/) (2025)
- [AbstractDifferentiation.jl: Backend-Agnostic Differentiable Programming in Julia](https://mlsanity.com/paper/2109.12449/) (2021)
- [Dressi: A Hardware-Agnostic Differentiable Renderer with Reactive Shader Packing and Soft Rasterization](https://mlsanity.com/paper/2204.01386/) (2022)
- [Machines of finite depth: towards a formalization of neural networks](https://mlsanity.com/paper/2204.12786/) (2022)
- [Optimizing Automatic Differentiation with Deep Reinforcement Learning](https://mlsanity.com/paper/2406.05027/) (2024)
- [Understanding Automatic Differentiation Pitfalls](https://mlsanity.com/paper/2305.07546/) (2023)
- [JAXbind: Bind any function to JAX](https://mlsanity.com/paper/2403.08847/) (2024)
- [Kolmogorov-Arnold Energy Models: Fast, Interpretable Generative Modeling](https://mlsanity.com/paper/2506.14167/) (2025)
- [DaCe AD: Unifying High-Performance Automatic Differentiation for Machine Learning and Scientific Computing](https://mlsanity.com/paper/2509.02197/) (2025)
- [Stochastic Sample Approximations of (Local) Moduli of Continuity](https://mlsanity.com/paper/2509.15368/) (2025)
- [NORi: An ML-Augmented Ocean Boundary Layer Parameterization](https://mlsanity.com/paper/2512.04452/) (2025)
- [Deep Adaptive Model-Based Design of Experiments](https://mlsanity.com/paper/2603.16146/) (2026)
- [Bidirectional Reactive Programming for Machine Learning](https://mlsanity.com/paper/2311.16977/) (2023)
- [Action-Gradient Monte Carlo Tree Search for Non-Parametric Continuous (PO)MDPs](https://mlsanity.com/paper/2503.12181/) (2025)
- [A Common Interface for Automatic Differentiation](https://mlsanity.com/paper/2505.05542/) (2025)
- [Large-scale empirical tuning and comparison of default optimizers for variational inference](https://mlsanity.com/paper/2606.07841/) (2026)
- [Stable and Scalable Probabilistic Numerical Solvers for Stiff and High-Dimensional ODEs](https://mlsanity.com/paper/2606.08203/) (2026)
- [Mosaic: A Benchmark Suite for Differentiable Physics Solvers](https://mlsanity.com/paper/2606.27895/) (2026)
- [Autonomous phase discovery](https://mlsanity.com/paper/2609.33802/) (2026)
- [Recursive Improvement of a Differentiable Scientific Software Ecosystem](https://mlsanity.com/paper/2610.04561/) (2026)

## Abstract (arXiv)

Applying differentiable programming techniques and machine learning algorithms to foreign programs requires developers to either rewrite their code in a machine learning framework, or otherwise provide derivatives of the foreign code. This paper presents Enzyme, a high-performance automatic differentiation (AD) compiler plugin for the LLVM compiler framework capable of synthesizing gradients of statically analyzable programs expressed in the LLVM intermediate representation (IR). Enzyme synthesizes gradients for programs written in any language whose compiler targets LLVM IR including C, C++, Fortran, Julia, Rust, Swift, MLIR, etc., thereby providing native AD capabilities in these languages. Unlike traditional source-to-source and operator-overloading tools, Enzyme performs AD on optimized IR. On a machine-learning focused benchmark suite including Microsoft's ADBench, AD on optimized IR achieves a geometric mean speedup of 4.5x over AD on IR before optimization allowing Enzyme to achieve state-of-the-art performance. Packaging Enzyme for PyTorch and TensorFlow provides convenient access to gradients of foreign code with state-of-the art performance, enabling foreign code to be directly incorporated into existing machine learning workflows.