How Much of a Real Workload Can LLM-Generated GPU Kernels Actually Reach?
Not on the map. Why?
Abstract (arXiv)
Language models can now write GPU kernels that outperform PyTorch. We evaluate five model configurations on KernelBench level 1 and find that a frontier model produces correct kernels for 91.1% of problems and independently verified speedups on 22 of 56, including three convolutions, with a median of 1.235x. Open-weights models are far behind: the best reaches 30.4% correct with three verified speedups and solves zero convolutions. We then ask a question the literature does not: what fraction of a real model's wall clock do such kernels govern? Profiling seven workloads across three domains, we find the addressable fraction ranges from 8.9% to 58.2%. On transformers, 80-86% of runtime is spent in cuBLAS GEMM and FlashAttention, bounding realistic end-to-end improvement at roughly 1%, and the fraction shrinks with model scale. On recommenders it is 58.2%, concentrated in a single embedding kernel. We introduce DLRM-Bench, 12 recommender kernel problems in KernelBench format, and measure a 41.7% win rate at a 1.552x median there, projecting 8.63% end-to-end. Separately, we show that KernelBench's correctness check (torch.allclose with an absolute tolerance) is satisfied by a tensor of zeros on 4 of 60 level-1 problems. Two kernels in our own results exploited this before we detected them, including one scored at 283x that wrote 0.3% of its output buffer. We propose scale-invariant replacements and release all 879 evaluations.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 7 of its 15 references that are in the map. First posted on arXiv 17 Sep 2026.
Builds on
7 of 15 references.
References · 7
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
- 2019Deep Learning Recommendation Model for Personalization and Recommendation Systemscited by 327
- 2025KernelBench: Can LLMs Write Efficient GPU Kernels?cited by 225
- 2025GPU Kernel Scientist: An LLM-Driven Framework for Iterative Kernel Optimizationcited by 22
- 2025ConCuR: Conciseness Makes State-of-the-Art Kernel Generationcited by 13
- 2026KernelBenchX: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernelscited by 12
- 2026Correct but Slow: An Empirical Study of the GPU Kernel Evaluation Gap in Modern Domain-Specific Languagescited by 1
Built on by
Not cited yet.