# The Correctness Illusion in LLM-Generated GPU Kernels

arXiv: https://arxiv.org/abs/2606.20128 · HTML: https://mlsanity.com/paper/2606.20128/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 18 references that are in the map.
- First posted on arXiv 18 Jun 2026.

## Builds on (most cited first, 12 of 12)

- [A Study of BFLOAT16 for Deep Learning Training](https://mlsanity.com/paper/1905.12322/) (2019)
- [KernelBench: Can LLMs Write Efficient GPU Kernels?](https://mlsanity.com/paper/2502.10517/) (2025)
- [Geak: Introducing Triton Kernel AI Agent & Evaluation Benchmarks](https://mlsanity.com/paper/2507.23194/) (2025)
- [Free Lunch for Testing: Fuzzing Deep-Learning Libraries from Open Source](https://mlsanity.com/paper/2201.06589/) (2022)
- [STARK: Strategic Team of Agents for Refining Kernels](https://mlsanity.com/paper/2510.16996/) (2025)
- [DocTer: Documentation Guided Fuzzing for Testing Deep Learning API Functions](https://mlsanity.com/paper/2109.01002/) (2021)
- [Fuzzing Automatic Differentiation in Deep-Learning Libraries](https://mlsanity.com/paper/2302.04351/) (2023)
- [Fuzzing Deep-Learning Libraries via Automated Relational API Inference](https://mlsanity.com/paper/2207.05531/) (2022)
- [KernelBenchX: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernels](https://mlsanity.com/paper/2605.04956/) (2026)
- [KernelBand: Steering LLM-based Kernel Optimization via Hardware-Aware Multi-Armed Bandits](https://mlsanity.com/paper/2511.18868/) (2025)
- [Benchmarking Deep Learning Fuzzers](https://mlsanity.com/paper/2310.06912/) (2023)
- [GPU-Fuzz: Finding Memory Errors in Deep Learning Frameworks](https://mlsanity.com/paper/2602.10478/) (2026)

## Built on by (most cited first, 6 of 6)

- [Operator-Aware Mixed-Precision Tolerance Calibration for Tensor Kernels](https://mlsanity.com/paper/2607.16228/) (2026)
- [Test-Input Generation for Tensor Programs: What Actually Finds Kernel Bugs](https://mlsanity.com/paper/2606.27396/) (2026)
- [Don't Regenerate, Debug: A Domain-Specific Agent for Repairing Near-Miss Hardware Operators](https://mlsanity.com/paper/2608.02712/) (2026)
- [A Contract-Grade Verifier for LLM-Generated GPU Kernels, and a Native Blackwell Backward for the Gated-Linear-Recurrence Family](https://mlsanity.com/paper/2608.12700/) (2026)
- [HIERA: Workload-Aware Planning Across Implementation Spaces for GPU Kernel Optimization](https://mlsanity.com/paper/2608.21157/) (2026)
- [Measuring the Checker: Mutation Analysis for GPU-Kernel Benchmark Oracles](https://mlsanity.com/paper/2609.22220/) (2026)

## Abstract (arXiv)

Benchmarks for LLM-generated GPU kernels (KernelBench, TritonBench, GEAK) score correctness through fixed-shape, small-sample allclose-style checks. The number of inputs varies between benchmarks. The shape, dtype, and tolerance are fixed for each kernel. We test that oracle empirically. We construct a controlled corpus of 24 Triton and CPU stand-in kernels (15 correct controls and 9 LLM-style buggy variants seeded with documented transcription errors) and re-evaluate it under op-schema-aware seeded fuzzing with a high-precision (fp64) CPU reference and per-(op, dtype) absolute tolerances. The seeded oracle flags 9 of 9 buggy kernels and passes 15 of 15 correct controls, at zero precision cost on controls. We extend the corpus to 26 ops (adding a flash-attention pair) and re-run the same protocol on five GPU classes (RTX 3060, A10, L40S, A100 SXM4, H100 NVL). The verdicts are identical across all five GPUs: 10 of 10 illusions caught and 16 of 16 controls clean. The corpus result is about LLM-style transcription bugs that the allclose-on-one-shape oracle certifies as correct, not about the bug rate of any specific deployed LLM. Every flagged failure replays byte-for-byte from a stored seed.