# nablaSD: Differentiable Programming for Sparse Tensors

arXiv: https://arxiv.org/abs/2303.07030 · HTML: https://mlsanity.com/paper/2303.07030/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 48 references that are in the map.
- First posted on arXiv 13 Mar 2023.

## Builds on (most cited first, 14 of 14)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [The simple essence of automatic differentiation](https://mlsanity.com/paper/1804.00746/) (2018)
- [Format Abstraction for Sparse Tensor Algebra Compilers](https://mlsanity.com/paper/1804.10112/) (2018)
- [Demystifying Differentiable Programming: Shift/Reset the Penultimate Backpropagator](https://mlsanity.com/paper/1803.10228/) (2018)
- [Correctness of Automatic Differentiation via Diffeologies and Categorical Gluing](https://mlsanity.com/paper/2001.02209/) (2020)
- [Getting to the Point. Index Sets and Parallelism-Preserving Autodiff for Pointful Array Programming](https://mlsanity.com/paper/2104.05372/) (2021)
- [You Only Linearize Once: Tangents Transpose to Gradients](https://mlsanity.com/paper/2204.10923/) (2022)
- [Optimizing Tensor Programs on Flexible Storage](https://mlsanity.com/paper/2210.06267/) (2022)
- [Efficient Differentiable Programming in a Functional Array-Processing Language](https://mlsanity.com/paper/1806.02136/) (2018)
- [Functional Collection Programming with Semi-Ring Dictionaries](https://mlsanity.com/paper/2103.06376/) (2021)
- [Optimized Sparse Matrix Operations for Reverse Mode Automatic Differentiation](https://mlsanity.com/paper/2212.05159/) (2022)
- [Forward- or Reverse-Mode Automatic Differentiation: What's the Difference?](https://mlsanity.com/paper/2212.11088/) (2022)
- [Parallel Dual-Numbers Reverse AD](https://mlsanity.com/paper/2207.03418/) (2022)
- [Denotationally Correct, Purely Functional, Efficient Reverse-mode Automatic Differentiation](https://mlsanity.com/paper/2212.09801/) (2022)

## Abstract (arXiv)

Sparse tensors are prevalent in many data-intensive applications, yet existing differentiable programming frameworks are tailored towards dense tensors. This presents a significant challenge for efficiently computing gradients through sparse tensor operations, as their irregular sparsity patterns can result in substantial memory and computational overheads. In this work, we introduce a novel framework that enables the efficient and automatic differentiation of sparse tensors, addressing this fundamental issue. Our experiments demonstrate the effectiveness of the proposed framework in terms of performance and scalability, outperforming state-of-the-art frameworks across a range of synthetic and real-world datasets. Our approach offers a promising direction for enabling efficient and scalable differentiable programming with sparse tensors, which has significant implications for numerous applications in machine learning, natural language processing, and scientific computing.