# Accelerating a Triton Fused Kernel for W4A16 Quantized Inference with SplitK work decomposition

arXiv: https://arxiv.org/abs/2402.00025 · HTML: https://mlsanity.com/paper/2402.00025/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 1 of its 7 references that are in the map.
- First posted on arXiv 5 Jan 2024.

## Builds on (most cited first, 1 of 1)

- [Stream-K: Work-centric Parallel Decomposition for Dense Matrix-Matrix Multiplication on the GPU](https://mlsanity.com/paper/2301.03598/) (2023)

## Built on by (most cited first, 4 of 4)

- [LLM-42: Enabling Determinism in LLM Inference with Verified Speculation](https://mlsanity.com/paper/2601.17768/) (2026)
- [Tilus: A Tile-Level GPGPU Programming Language for Low-Precision Computation](https://mlsanity.com/paper/2504.12984/) (2025)
- [HASTE: Hardware-Aware Dynamic Sparse Training for Large Output Spaces](https://mlsanity.com/paper/2606.01117/) (2026)
- [Celty: SpMSpV GPU Kernel and SIMT Co-Design for Efficient Dual-Sparse LLM Inference](https://mlsanity.com/paper/2608.01536/) (2026)

## Abstract (arXiv)

We propose an implementation of an efficient fused matrix multiplication kernel for W4A16 quantized inference, where we perform dequantization and GEMM in a fused kernel using a SplitK work decomposition. Our implementation shows improvement for the type of skinny matrix-matrix multiplications found in foundation model inference workloads. In particular, this paper surveys the type of matrix multiplication between a skinny activation matrix and a square weight matrix. Our results show an average of 65% speed improvement on A100, and an average of 124% speed improvement on H100 (with a peak of 295%) for a range of matrix dimensions including those found in a llama-style model, where m < n = k.