mlsanityData to 7 Oct 2026

The Correctness Illusion in LLM-Generated GPU Kernels

arXiv 2606.20128 ↗First posted 18 Jun 2026cs.SE cs.DC cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Benchmarks for LLM-generated GPU kernels (KernelBench, TritonBench, GEAK) score correctness through fixed-shape, small-sample allclose-style checks. The number of inputs varies between benchmarks. The shape, dtype, and tolerance are fixed for each kernel. We test that oracle empirically. We construct a controlled corpus of 24 Triton and CPU stand-in kernels (15 correct controls and 9 LLM-style buggy variants seeded with documented transcription errors) and re-evaluate it under op-schema-aware seeded fuzzing with a high-precision (fp64) CPU reference and per-(op, dtype) absolute tolerances. The seeded oracle flags 9 of 9 buggy kernels and passes 15 of 15 correct controls, at zero precision cost on controls. We extend the corpus to 26 ops (adding a flash-attention pair) and re-run the same protocol on five GPU classes (RTX 3060, A10, L40S, A100 SXM4, H100 NVL). The verdicts are identical across all five GPUs: 10 of 10 illusions caught and 16 of 16 controls clean. The corpus result is about LLM-style transcription bugs that the allclose-on-one-shape oracle certifies as correct, not about the bug rate of any specific deployed LLM. Every flagged failure replays byte-for-byte from a stored seed.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 12 of its 18 references that are in the map. First posted on arXiv 18 Jun 2026.

Builds on

12 of 18 references.

References · 12

  1. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 227
  2. 2025KernelBench: Can LLMs Write Efficient GPU Kernels?cited by 206
  3. 2025Geak: Introducing Triton Kernel AI Agent & Evaluation Benchmarkscited by 31
  4. 2022Free Lunch for Testing: Fuzzing Deep-Learning Libraries from Open Sourcecited by 27
  5. 2025STARK: Strategic Team of Agents for Refining Kernelscited by 23
  6. 2021DocTer: Documentation Guided Fuzzing for Testing Deep Learning API Functionscited by 21
  7. 2022Fuzzing Deep-Learning Libraries via Automated Relational API Inferencecited by 16
  8. 2023Fuzzing Automatic Differentiation in Deep-Learning Librariescited by 16
  9. 2026KernelBenchX: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernelscited by 10
  10. 2025KernelBand: Steering LLM-based Kernel Optimization via Hardware-Aware Multi-Armed Banditscited by 2
  11. 2023Benchmarking Deep Learning Fuzzerscited by 1
  12. 2026GPU-Fuzz: Finding Memory Errors in Deep Learning Frameworkscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Operator-Aware Mixed-Precision Tolerance Calibration for Tensor Kernelscited by 1
  2. 2026Test-Input Generation for Tensor Programs: What Actually Finds Kernel Bugscited by 0
  3. 2026Don't Regenerate, Debug: A Domain-Specific Agent for Repairing Near-Miss Hardware Operatorscited by 0
  4. 2026A Contract-Grade Verifier for LLM-Generated GPU Kernels, and a Native Blackwell Backward for the Gated-Linear-Recurrence Familycited by 0
  5. 2026HIERA: Workload-Aware Planning Across Implementation Spaces for GPU Kernel Optimizationcited by 0
  6. 2026Measuring the Checker: Mutation Analysis for GPU-Kernel Benchmark Oraclescited by 0