mlsanityData to 7 Oct 2026

Optimized Sparse Matrix Operations for Reverse Mode Automatic Differentiation

arXiv 2212.05159 ↗First posted 10 Dec 2022cs.LG cs.MS cs.NAcited by 5

Not on the map. Why?

Abstract (arXiv)

Sparse matrix representations are ubiquitous in computational science and machine learning, leading to significant reductions in compute time, in comparison to dense representation, for problems that have local connectivity. The adoption of sparse representation in leading ML frameworks such as PyTorch is incomplete, however, with support for both automatic differentiation and GPU acceleration missing. In this work, we present an implementation of a CSR-based sparse matrix wrapper for PyTorch with CUDA acceleration for basic matrix operations, as well as automatic differentiability. We also present several applications of the resulting sparse kernels to optimization problems, demonstrating ease of implementation and performance measurements versus their dense counterparts.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 15 of its 61 references that are in the map. First posted on arXiv 10 Dec 2022.

Builds on

15 of 61 references.

References · 15

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  4. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
  5. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,414
  6. 2013Spectral Networks and Locally Connected Networks on Graphscited by 1,097
  7. 2016Revisiting Semi-Supervised Learning with Graph Embeddingscited by 970
  8. 2015Automatic differentiation in machine learning: a surveycited by 846
  9. 2019GraphSAINT: Graph Sampling Based Inductive Learning Methodcited by 618
  10. 2019Learning to Optimize Multigrid PDE Solverscited by 79
  11. 2020Learning Algebraic Multigrid Using Graph Neural Networkscited by 54
  12. 2022Learning Interface Conditions in Domain Decomposition Solverscited by 11
3 more
  1. 2020Tuning Multigrid Methods with Robust Optimizationcited by 2
  2. 2019GLU3.0: Fast GPU-based Parallel Sparse LU Factorization for Circuit Simulationcited by 1
  3. 2020GSoFa: Scalable Sparse Symbolic LU Factorization on GPUscited by 1

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 2.

5 papers.

  1. 2023Neural incomplete factorization: learning preconditioners for the conjugate gradient methodcited by 18
  2. 2023MG-GNN: Multigrid Graph Neural Networks for Learning Multilevel Domain Decomposition Methodscited by 12
  3. 2024Learning incomplete factorization preconditioners for GMREScited by 9
  4. 2023nablaSD: Differentiable Programming for Sparse Tensorscited by 0
  5. 2024Deep Learning-Enhanced Preconditioning for Efficient Conjugate Gradient Solvers in Large-Scale PDE Systemscited by 0