# Compilation of Modular and General Sparse Workspaces

arXiv: https://arxiv.org/abs/2404.04541 · HTML: https://mlsanity.com/paper/2404.04541/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Apr 2024.

## Built on by (most cited first, 4 of 4)

- [Scorch: A Library for Sparse Deep Learning](https://mlsanity.com/paper/2405.16883/) (2024)
- [Ember: A Compiler for Efficient Embedding Operations on Decoupled Access-Execute Architectures](https://mlsanity.com/paper/2504.09870/) (2025)
- [Morphing-based Compression for Data-centric ML Pipelines](https://mlsanity.com/paper/2504.11067/) (2025)
- [FuseFlow: A Fusion-Centric Compilation Framework for Sparse Deep Learning on Streaming Dataflow](https://mlsanity.com/paper/2511.04768/) (2025)

## Abstract (arXiv)

Recent years have seen considerable work on compiling sparse tensor algebra expressions. This paper addresses a shortcoming in that work, namely how to generate efficient code (in time and space) that scatters values into a sparse result tensor. We address this shortcoming through a compiler design that generates code that uses sparse intermediate tensors (sparse workspaces) as efficient adapters between compute code that scatters and result tensors that do not support random insertion. Our compiler automatically detects sparse scattering behavior in tensor expressions and inserts necessary intermediate workspace tensors. We present an algorithm template for workspace insertion that is the backbone of our code generation algorithm. Our algorithm template is modular by design, supporting sparse workspaces that span multiple user-defined implementations. Our evaluation shows that sparse workspaces can be up to 27.12× faster than the dense workspaces of prior work. On the other hand, dense workspaces can be up to 7.58× faster than the sparse workspaces generated by our compiler in other situations, which motivates our compiler design that supports both. Our compiler produces sequential code that is competitive with hand-optimized linear and tensor algebra libraries on the expressions they support, but that generalizes to any other expression. Sparse workspaces are also more memory efficient than dense workspaces as they compress away zeros. This compression can asymptotically decrease memory usage, enabling tensor computations on data that would otherwise run out of memory.