A Contract-Grade Verifier for LLM-Generated GPU Kernels, and a Native Blackwell Backward for the Gated-Linear-Recurrence Family
Not on the map. Why?
Abstract (arXiv)
Systems that generate GPU kernels with language models report high correctness rates. Those rates come from a single loose test: run the kernel on a few random inputs at one fixed shape and accept it if the output is close to a reference. A kernel can pass that test and still be silently wrong. It can return an ordinary number where the true answer is a NaN or an infinity, differ from run to run, break when the shape changes, or accumulate in fp16 where the reference keeps an fp32 total. We build the instrument that checks correctness properly: a contract-grade verifier of twelve adversarial gates, each a property a correct kernel must satisfy, several of them tolerance-free, so no choice of threshold can explain a failure away. Aimed outward, the verifier audits 2,638 machine-generated kernels that a public system's own harness had already accepted as correct. It finds 39.5% broken beyond any tolerance argument and 62.1% carrying at least one violation. The field's standard test accepts 1,487 kernels the verifier rejects, against only 14 the other way. We defend the finding four independent ways: a 7/7 positive control, a threshold-calibration sweep, 98.5% agreement with the reference benchmark's own correctness code, and a stratified hand-audit. Aimed inward, the verifier judges a kernel of our own: the first native Blackwell tcgen05 training backward for the gated-linear-recurrence (GDN) family, including the reverse-state stage the field still runs on a fallback. We establish its correctness independently, against a double-precision oracle, and train five family members through it. The correctness signal behind reported progress in kernel generation is far weaker than the numbers suggest, and a set of tolerance-free contracts would close most of the gap.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 16 of its 22 references that are in the map. First posted on arXiv 13 Aug 2026.
Builds on
16 of 22 references.
References · 16
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
- 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
- 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
- 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
- 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
- 2025KernelBench: Can LLMs Write Efficient GPU Kernels?cited by 225
- 2026Mamba-3: Improved Sequence Modeling using State Space Principlescited by 89
- 2025TritonBench: Benchmarking Large Language Model Capabilities for Generating Triton Operatorscited by 75
- 2025CUDA-L1: Improving CUDA Optimization via Contrastive Reinforcement Learningcited by 61
- 2025Towards Robust Agentic CUDA Kernel Benchmarking, Verification, and Optimizationcited by 56
- 2026Dr. Kernel: Reinforcement Learning Done Right for Triton Kernel Generationscited by 22
4 more
- 2026Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loopscited by 9
- 2026The Correctness Illusion in LLM-Generated GPU Kernelscited by 6
- 2026Kernel Contracts: A Specification Language for ML Kernel Correctness Across Heterogeneous Siliconcited by 3
- 2026Operator-Aware Mixed-Precision Tolerance Calibration for Tensor Kernelscited by 1
Built on by
Not cited yet.