# Kernel Operations on the GPU, with Autodiff, without Memory Overflows

arXiv: https://arxiv.org/abs/2004.11127 · HTML: https://mlsanity.com/paper/2004.11127/

## Key facts

- Cited by 104 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Mar 2020.

## Built on by (most cited first, 30 of 104)

- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Diagonal State Spaces are as Effective as Structured State Spaces](https://mlsanity.com/paper/2203.14343/) (2022)
- [Sinkformers: Transformers with Doubly Stochastic Attention](https://mlsanity.com/paper/2110.11773/) (2021)
- [Liquid Structural State-Space Models](https://mlsanity.com/paper/2209.12951/) (2022)
- [Shape registration in the time of transformers](https://mlsanity.com/paper/2106.13679/) (2021)
- [Accurate Point Cloud Registration with Robust Optimal Transport](https://mlsanity.com/paper/2111.00648/) (2021)
- [On UMAP's true loss function](https://mlsanity.com/paper/2103.14608/) (2021)
- [Towards a mathematical theory of trajectory inference](https://mlsanity.com/paper/2102.09204/) (2021)
- [From t-SNE to UMAP with contrastive learning](https://mlsanity.com/paper/2206.01816/) (2022)
- [Posterior and Computational Uncertainty in Gaussian Processes](https://mlsanity.com/paper/2205.15449/) (2022)
- [Preconditioning for Scalable Gaussian Process Hyperparameter Optimization](https://mlsanity.com/paper/2107.00243/) (2021)
- [Latent Graph Inference using Product Manifolds](https://mlsanity.com/paper/2211.16199/) (2022)
- [Simplifying and Understanding State Space Models with Diagonal Linear RNNs](https://mlsanity.com/paper/2212.00768/) (2022)
- [Tighter Bounds on the Log Marginal Likelihood of Gaussian Process Regression Using Conjugate Gradients](https://mlsanity.com/paper/2102.08314/) (2021)
- [Elastic shape analysis of surfaces with second-order Sobolev metrics: a comprehensive numerical framework](https://mlsanity.com/paper/2204.04238/) (2022)
- [Decoupled Exploration and Exploitation Policies for Sample-Efficient Reinforcement Learning](https://mlsanity.com/paper/2101.09458/) (2021)
- [Memory-Scalable and Simplified Functional Map Learning](https://mlsanity.com/paper/2404.00330/) (2024)
- [ResNet-LDDMM: Advancing the LDDMM Framework using Deep Residual Networks](https://mlsanity.com/paper/2102.07951/) (2021)
- [Fast and Scalable Optimal Transport for Brain Tractograms](https://mlsanity.com/paper/2107.02010/) (2021)
- [Fast Nonlinear Vector Quantile Regression](https://mlsanity.com/paper/2205.14977/) (2022)
- [Efficient Hyperparameter Tuning for Large Scale Kernel Ridge Regression](https://mlsanity.com/paper/2201.06314/) (2022)
- [qPOTS: Efficient batch multiobjective Bayesian optimization via Pareto optimal Thompson sampling](https://mlsanity.com/paper/2310.15788/) (2023)
- [Large-Scale Gaussian Processes via Alternating Projection](https://mlsanity.com/paper/2310.17137/) (2023)
- [FlashSinkhorn: IO-Aware Entropic Optimal Transport on GPU](https://mlsanity.com/paper/2602.03067/) (2026)
- [Low-Precision Arithmetic for Fast Gaussian Processes](https://mlsanity.com/paper/2207.06856/) (2022)
- [ELECTRA: A Cartesian Network for 3D Charge Density Prediction with Floating Orbitals](https://mlsanity.com/paper/2503.08305/) (2025)
- [SKIing on Simplices: Kernel Interpolation on the Permutohedral Lattice for Scalable Gaussian Processes](https://mlsanity.com/paper/2106.06695/) (2021)
- [Towards Efficient Point Cloud Graph Neural Networks Through Architectural Simplification](https://mlsanity.com/paper/2108.06317/) (2021)
- [Conditional simulation via entropic optimal transport: Toward non-parametric estimation of conditional Brenier maps](https://mlsanity.com/paper/2411.07154/) (2024)
- [Position: Curvature Matrices Should Be Democratized via Linear Operators](https://mlsanity.com/paper/2501.19183/) (2025)

## Abstract (arXiv)

The KeOps library provides a fast and memory-efficient GPU support for tensors whose entries are given by a mathematical formula, such as kernel and distance matrices. KeOps alleviates the major bottleneck of tensor-centric libraries for kernel and geometric applications: memory consumption. It also supports automatic differentiation and outperforms standard GPU baselines, including PyTorch CUDA tensors or the Halide and TVM libraries. KeOps combines optimized C++/CUDA schemes with binders for high-level languages: Python (Numpy and PyTorch), Matlab and GNU R. As a result, high-level "quadratic" codes can now scale up to large data sets with millions of samples processed in seconds. KeOps brings graphics-like performances for kernel methods and is freely available on standard repositories (PyPi, CRAN). To showcase its versatility, we provide tutorials in a wide range of settings online at urlwww.kernel-operations.io.