# Sparser, Better, Faster, Stronger: Sparsity Detection for Efficient Automatic Differentiation

arXiv: https://arxiv.org/abs/2501.17737 · HTML: https://mlsanity.com/paper/2501.17737/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 95 references that are in the map.
- First posted on arXiv 29 Jan 2025.

## Builds on (most cited first, 21 of 21)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability](https://mlsanity.com/paper/2103.00065/) (2021)
- [Universal Differential Equations for Scientific Machine Learning](https://mlsanity.com/paper/2001.04385/) (2020)
- [A Conceptual Introduction to Hamiltonian Monte Carlo](https://mlsanity.com/paper/1701.02434/) (2017)
- [Efficient and Modular Implicit Differentiation](https://mlsanity.com/paper/2105.15183/) (2021)
- [Implicit Graph Neural Networks](https://mlsanity.com/paper/2009.06211/) (2020)
- [A Differentiable Programming System to Bridge Machine Learning and Scientific Computing](https://mlsanity.com/paper/1907.07587/) (2019)
- [The Power Grid Library for Benchmarking AC Optimal Power Flow Algorithms](https://mlsanity.com/paper/1908.02788/) (2019)
- [An Introduction to Probabilistic Programming](https://mlsanity.com/paper/1809.10756/) (2018)
- [Fashionable Modelling with Flux](https://mlsanity.com/paper/1811.01457/) (2018)
- [JuMP: A Modeling Language for Mathematical Optimization](https://mlsanity.com/paper/1508.01982/) (2015)
- [Don't Unroll Adjoint: Differentiating SSA-Form Programs](https://mlsanity.com/paper/1810.07951/) (2018)
- [The Elements of Differentiable Programming](https://mlsanity.com/paper/2403.14606/) (2024)
- [Instead of Rewriting Foreign Code for Machine Learning, Automatically Synthesize Fast Gradients](https://mlsanity.com/paper/2010.01709/) (2020)
- [CANOS: A Fast and Scalable Neural AC-OPF Solver Robust To N-1 Perturbations](https://mlsanity.com/paper/2403.17660/) (2024)
- [Accelerating Optimal Power Flow with GPUs: SIMD Abstraction of Nonlinear Programs and Condensed-Space Interior-Point Methods](https://mlsanity.com/paper/2307.16830/) (2023)
- [High-performance symbolic-numerics via multiple dispatch](https://mlsanity.com/paper/2105.03949/) (2021)
- [A new framework for the computation of Hessians](https://mlsanity.com/paper/2007.15040/) (2020)
- [Optimizing Automatic Differentiation with Deep Reinforcement Learning](https://mlsanity.com/paper/2406.05027/) (2024)

## Built on by (most cited first, 4 of 4)

- [A Common Interface for Automatic Differentiation](https://mlsanity.com/paper/2505.05542/) (2025)
- [NaN-Propagation: A Novel Method for Sparsity Detection in Black-Box Computational Functions](https://mlsanity.com/paper/2507.23186/) (2025)
- [Scalable Bayesian Inference for Nonlinear Conservation Laws](https://mlsanity.com/paper/2605.31127/) (2026)
- [Truncated automatic sparse differentiation for machine learning interatomic potentials](https://mlsanity.com/paper/2609.20510/) (2026)

## Abstract (arXiv)

From implicit differentiation to probabilistic modeling, Jacobian and Hessian matrices have many potential use cases in Machine Learning (ML), but they are viewed as computationally prohibitive. Fortunately, these matrices often exhibit sparsity, which can be leveraged to speed up the process of Automatic Differentiation (AD). This paper presents advances in sparsity detection, previously the performance bottleneck of Automatic Sparse Differentiation (ASD). Our implementation of sparsity detection is based on operator overloading, able to detect both local and global sparsity patterns, and supports flexible index set representations. It is fully automatic and requires no modification of user code, making it compatible with existing ML codebases. Most importantly, it is highly performant, unlocking Jacobians and Hessians at scales where they were considered too expensive to compute. On real-world problems from scientific ML, graph neural networks and optimization, we show significant speed-ups of up to three orders of magnitude. Notably, using our sparsity detection system, ASD outperforms standard AD for one-off computations, without amortization of either sparsity detection or matrix coloring.