mlsanityData to 7 Oct 2026

SparseTIR: Composable Abstractions for Sparse Compilation in Deep Learning

arXiv 2207.04606 ↗First posted 11 Jul 2022cs.LG cs.AI cs.PLcited by 47

Not on the map. Why?

Abstract (arXiv)

Sparse tensors are rapidly becoming critical components of modern deep learning workloads. However, developing high-performance sparse operators can be difficult and tedious, and existing vendor libraries cannot satisfy the escalating demands from new operators. Sparse tensor compilers simplify the development of operators, but efficient sparse compilation for deep learning remains challenging because a single sparse format cannot maximize hardware efficiency, and single-shot compilers cannot keep up with latest hardware and system advances. In this paper, we observe that the key to addressing both these challenges is to leverage composable formats and composable transformations. We propose SparseTIR, a sparse tensor compilation abstraction that offers composable formats and composable transformations for deep learning workloads. SparseTIR constructs a search space over these composable components for performance tuning. With these improvements, SparseTIR obtains consistent performance speedups vs vendor libraries on GPUs for single operators: 1.20-2.34x for GNN operators, 1.05-2.98x for sparse attention operators, and 0.56-7.45x for sparse convolution operators. SparseTIR also accelerates end-to-end GNNs by 1.08-1.52x for GraphSAGE training, and 4.20-40.18x for RGCN inference.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 50 of its 121 references that are in the map. First posted on arXiv 11 Jul 2022.

Builds on

50 of 121 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  4. 2017Graph Attention Networkscited by 8,854
  5. 2017Inductive Representation Learning on Large Graphscited by 6,284
  6. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,414
  11. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  12. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
38 more
  1. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  2. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,684
  3. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  4. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  5. 2019Heterogeneous Graph Attention Networkcited by 732
  6. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
  7. 2020Heterogeneous Graph Transformercited by 581
  8. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  9. 2018Hypergraph Neural Networkscited by 482
  10. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  11. 2020Ansor: Generating High-Performance Tensor Programs for Deep Learningcited by 199
  12. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 177
  13. 2021Block Pruning For Faster Transformerscited by 176
  14. 2018Learning to Optimize Tensor Programscited by 138
  15. 2020Sparse GPU Kernels for Deep Learningcited by 138
  16. 2021Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Modelscited by 75
  17. 2022TorchSparse: Efficient Point Cloud Inference Enginecited by 72
  18. 2020GNNAdvisor: An Adaptive and Efficient Runtime System for GNN Acceleration on GPUscited by 51
  19. 2019SciPy 1.0--Fundamental Algorithms for Scientific Computing in Pythoncited by 49
  20. 2020GE-SpMM: General-purpose Sparse Matrix-Matrix Multiplication on GPUs for Graph Neural Networkscited by 42
  21. 2022TensorIR: An Abstraction for Automatic Tensorized Program Optimizationcited by 38
  22. 2020FeatGraph: A Flexible and Efficient Backend for Graph Neural Network Systemscited by 36
  23. 2022Tensor Program Optimization with Probabilistic Programscited by 30
  24. 2020FusedMM: A Unified SDDMM-SpMM Kernel for Graph Embedding and Graph Neural Networkscited by 26
  25. 2018Relay: A New IR for Machine Learning Frameworkscited by 24
  26. 2018Design Principles for Sparse Matrix Multiplication on the GPUcited by 18
  27. 2021QGTC: Accelerating Quantized Graph Neural Networks via GPU Tensor Corecited by 18
  28. 2020Automatic Horizontal Fusion for GPU Kernelscited by 17
  29. 2021The CoRa Tensor Compiler: Compilation for Ragged Tensors with Minimal Paddingcited by 17
  30. 2018Format Abstraction for Sparse Tensor Algebra Compilerscited by 16
  31. 2022Composable and Modular Code Generation in MLIR: A Structured and Retargetable Approach to Tensor Compiler Constructioncited by 8
  32. 2022Compiler Support for Sparse Tensor Computations in MLIRcited by 8
  33. 2020TIRAMISU: A Polyhedral Compiler for Dense and Sparse Deep Learningcited by 7
  34. 2022Heuristic Adaptability to Input Dynamics for SpMM on GPUscited by 6
  35. 2017Sympiler: Transforming Sparse Matrix Codes by Decoupling Symbolic Analysiscited by 4
  36. 2020Automatic Generation of Efficient Sparse Tensor Format Conversion Routinescited by 4
  37. 2021Accelerating SpMM Kernel with Cache-First Edge Sampling for Graph Neural Networkscited by 2
  38. 2022SparseLNR: Accelerating Sparse Tensor Computations Using Loop Nest Restructuringcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 13; 2024: 14; 2025: 13; 2026: 6.

47 papers.

Sort:

By area:

  1. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  2. 2025Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsitycited by 204
  3. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  4. 2025Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutationcited by 100
  5. 2023TorchSparse++: Efficient Training and Inference Framework for Sparse Convolution on GPUscited by 36
  6. 2023PIT: Optimization of Dynamic Sparse Deep Learning Models via Permutation Invariant Transformationcited by 25
  7. 2023A Survey on Graph Neural Network Acceleration: Algorithms, Systems, and Customized Hardwarecited by 18
  8. 2023Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsitycited by 14
  9. 2023Relax: Composable Abstractions for End-to-End Dynamic Machine Learningcited by 12
  10. 2023DyCL: Dynamic Neural Network Compilation Via Program Rewriting and Graph Optimizationcited by 9
  11. 2024Enabling Unstructured Sparse Acceleration on Structured Sparse Acceleratorscited by 8
  12. 2023GSplit: Scaling Graph Neural Network Training on Large Graphs via Split-Parallelismcited by 6
35 more
  1. 2025Your Compiler is Backdooring Your Model: Understanding and Exploiting Compilation Inconsistency Vulnerabilities in Deep Learning Compilerscited by 6
  2. 2023Cached Operator Reordering: A Unified View for Fast GNN Trainingcited by 5
  3. 2024SPLAT: A framework for optimised GPU code-generation for SParse reguLar ATtentioncited by 5
  4. 2024UniSparse: An Intermediate Language for General Sparse Format Customizationcited by 4
  5. 2026Event Tensor: A Unified Abstraction for Compiling Dynamic Megakernelcited by 4
  6. 2024GNNBENCH: Fair and Productive Benchmarking for Single-GPU GNN Systemcited by 3
  7. 2025PyGraph: Robust Compiler Support for CUDA Graphs in PyTorchcited by 3
  8. 2023Minuet: Accelerating 3D Sparse Convolutions on GPUscited by 2
  9. 2024PyGim: An Efficient Graph Neural Network Library for Real Processing-In-Memory Architecturescited by 2
  10. 2024DF-GNN: Dynamic Fusion Framework for Attention Graph Neural Networks on GPUscited by 2
  11. 2025Hexcute: A Compiler Framework for Automating Layout Synthesis in GPU Programscited by 2
  12. 2023STen: Productive and Efficient Sparsity in PyTorchcited by 1
  13. 2023SUBP: Soft Uniform Block Pruning for 1xN Sparse CNNs Multithreading Accelerationcited by 1
  14. 2024Single-GPU GNN Systems: Traps and Pitfallscited by 1
  15. 2024iSpLib: A Library for Accelerating Graph Neural Networks using Auto-tuned Sparse Operationscited by 1
  16. 2024Proteus: Preserving Model Confidentiality during Graph Optimizationscited by 1
  17. 2024Scorch: A Library for Sparse Deep Learningcited by 1
  18. 2024Optimizing Tensor Computation Graphs with Equality Saturation and Monte Carlo Tree Searchcited by 1
  19. 2025Libra: Unleashing GPU Heterogeneity for High-Performance Sparse Matrix Multiplicationcited by 1
  20. 2026Vortex: Efficient and Programmable Sparse Attention Serving for AI Agentscited by 1
  21. 2022ALT: Boosting Deep Learning Performance by Breaking the Wall between Graph and Operator Level Optimizationscited by 0
  22. 2023DLAS: An Exploration and Assessment of the Deep Learning Acceleration Stackcited by 0
  23. 2024GeoT: Tensor Centric Library for Graph Neural Network via Efficient Segment Reduction on GPUcited by 0
  24. 2024Fast Inference with Kronecker-Sparse Matricescited by 0
  25. 2024Compressing Structured Tensor Algebracited by 0
  26. 2025SparseTransX: Efficient Training of Translation-Based Knowledge Graph Embeddings Using Sparse Matrix Operationscited by 0
  27. 2025Ember: A Compiler for Efficient Embedding Operations on Decoupled Access-Execute Architecturescited by 0
  28. 2025COGNATE: Acceleration of Sparse Tensor Programs on Emerging Hardware using Transfer Learningcited by 0
  29. 2025FuseFlow: A Fusion-Centric Compilation Framework for Sparse Deep Learning on Streaming Dataflowcited by 0
  30. 2025DCC: Data-Centric Compilation of Machine Learning Kernels for Processing-In-Memory Architecturescited by 0
  31. 2025Spira: Exploiting Voxel Data Structural Properties for Efficient Sparse Convolution in Point Cloud Networkscited by 0
  32. 2026Automated Tensor-Relational Decomposition for Large-Scale Sparse Tensor Computationcited by 0
  33. 2026Accelerating GPU Inference of Large Language Models with Moderately Unstructured Sparse Weight Matricescited by 0
  34. 2026Unified Static-Dynamic Pruning for Efficient LLM Inferencecited by 0
  35. 2026SparseDitto: An Agentic Sparse Compilation Framework through Architecture-Aware Synthesis on GPUscited by 0