# Stream-K: Work-centric Parallel Decomposition for Dense Matrix-Matrix Multiplication on the GPU

arXiv: https://arxiv.org/abs/2301.03598 · HTML: https://mlsanity.com/paper/2301.03598/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jan 2023.

## Built on by (most cited first, 23 of 23)

- [FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision](https://mlsanity.com/paper/2407.08608/) (2024)
- [FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving](https://mlsanity.com/paper/2501.01005/) (2025)
- [FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusion](https://mlsanity.com/paper/2406.06858/) (2024)
- [MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models](https://mlsanity.com/paper/2408.11743/) (2024)
- [POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference](https://mlsanity.com/paper/2410.18038/) (2024)
- [Fast Matrix Multiplications for Lookup Table-Quantized LLMs](https://mlsanity.com/paper/2407.10960/) (2024)
- [Lean Attention: Hardware-Aware Scalable Attention Mechanism for the Decode-Phase of Transformers](https://mlsanity.com/paper/2405.10480/) (2024)
- [Relax: Composable Abstractions for End-to-End Dynamic Machine Learning](https://mlsanity.com/paper/2311.02103/) (2023)
- [Efficient and Adaptable Overlapping for Computation and Communication via Signaling and Reordering](https://mlsanity.com/paper/2504.19519/) (2025)
- [FlashAttention on a Napkin: A Diagrammatic Approach to Deep Learning IO-Awareness](https://mlsanity.com/paper/2412.03317/) (2024)
- [COMET: Towards Partical W4A4KV4 LLMs Serving](https://mlsanity.com/paper/2410.12168/) (2024)
- [GQSA: Group Quantization and Sparsity for Accelerating Large Language Model Inference](https://mlsanity.com/paper/2412.17560/) (2024)
- [Accelerating a Triton Fused Kernel for W4A16 Quantized Inference with SplitK work decomposition](https://mlsanity.com/paper/2402.00025/) (2024)
- [Boosting LLM Serving through Spatial-Temporal GPU Resource Sharing](https://mlsanity.com/paper/2504.19516/) (2025)
- [DASH: Deterministic Attention Scheduling for High-throughput Reproducible LLM Training](https://mlsanity.com/paper/2601.21824/) (2026)
- [Tilus: A Tile-Level GPGPU Programming Language for Low-Precision Computation](https://mlsanity.com/paper/2504.12984/) (2025)
- [SSSD: Simply-Scalable Speculative Decoding](https://mlsanity.com/paper/2411.05894/) (2024)
- [APT-LLM: Exploiting Arbitrary-Precision Tensor Core Computing for LLM Acceleration](https://mlsanity.com/paper/2508.19087/) (2025)
- [Design Space Exploration of DMA based Finer-Grain Compute Communication Overlap](https://mlsanity.com/paper/2512.10236/) (2025)
- [EGG: An Expert-Guided Agent Framework for Kernel Generation](https://mlsanity.com/paper/2606.26758/) (2026)
- [Analytical Resource Management for Fine-grained MoE Computation-Communication Overlap](https://mlsanity.com/paper/2609.07536/) (2026)
- [Hardware-Aware Features for CUTLASS Kernel Selection](https://mlsanity.com/paper/2609.35587/) (2026)
- [GPU Goldmines: Tunable WebGPU Kernels for Edge LLM Inference](https://mlsanity.com/paper/2610.17321/) (2026)

## Abstract (arXiv)

We introduce Stream-K, a work-centric parallelization of matrix multiplication (GEMM) and related computations in dense linear algebra. Whereas contemporary decompositions are primarily tile-based, our method operates by partitioning an even share of the aggregate inner loop iterations among physical processing elements. This provides a near-perfect utilization of computing resources, regardless of how efficiently the output tiling for any given problem quantizes across the underlying processing elements. On GPU processors, our Stream-K parallelization of GEMM produces a peak speedup of up to 14× and 6.7×, and an average performance response that is both higher and more consistent across 32,824 GEMM problem geometries than state-of-the-art math libraries such as CUTLASS and cuBLAS. Furthermore, we achieve this performance from a single tile size configuration per floating-point precision, whereas today's math libraries employ complex kernel-selection heuristics to select from a large ensemble of kernel variants.