mlsanityData to 14 Oct 2026

Test-Input Generation for Tensor Programs: What Actually Finds Kernel Bugs

arXiv 2606.27396 ↗First posted 23 Jun 2026cs.SE cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Test-input generation for tensor kernels is folkloric. Most projects pick a representative shape and dtype, run a fixed-shape allclose-style check, and ship. We make the choices explicit and measure them. Using the gpuemu op-schema-aware seeded fuzzer (arXiv:2606.20128), we evaluate seven test-generation strategies across a 26-op corpus (16 correct controls and 10 LLM-style buggy variants seeded with documented transcription patterns) on an RTX 3060 GPU instance. Strategies vary the shape candidate set, the dtype mix, and the input value distribution. We report each strategy on two axes: bug recall and control false-positive (FP) rate. Boundary-only shape sampling is the operationally safe winner: 78% recall on the 10 buggy kernels with 0% FP on the 16 controls. Adversarial value sampling reaches higher recall (99%) but inflates control FP to 94% because the strategy injects NaN and Inf inputs and the validator's NaN check fires on every kernel that propagates them, not only on buggy kernels. On the two softmax tail-mask bugs the "regular" strategy (no boundary shapes) catches 0%, while boundary raises recall to 100% and 62% respectively. That gap is the clearest single signal in the data. The corpus result is about which seeded bug patterns each strategy catches, not about the bug rate of any specific deployed LLM.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 11 of its 20 references that are in the map. First posted on arXiv 23 Jun 2026.

Builds on

11 of 20 references.

References · 11

  1. 2025KernelBench: Can LLMs Write Efficient GPU Kernels?cited by 225
  2. 2025Geak: Introducing Triton Kernel AI Agent & Evaluation Benchmarkscited by 32
  3. 2022Free Lunch for Testing: Fuzzing Deep-Learning Libraries from Open Sourcecited by 28
  4. 2025STARK: Strategic Team of Agents for Refining Kernelscited by 25
  5. 2021DocTer: Documentation Guided Fuzzing for Testing Deep Learning API Functionscited by 22
  6. 2023Fuzzing Automatic Differentiation in Deep-Learning Librariescited by 17
  7. 2022Fuzzing Deep-Learning Libraries via Automated Relational API Inferencecited by 16
  8. 2026KernelBenchX: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernelscited by 12
  9. 2026The Correctness Illusion in LLM-Generated GPU Kernelscited by 6
  10. 2025KernelBand: Steering LLM-based Kernel Optimization via Hardware-Aware Multi-Armed Banditscited by 2
  11. 2025Evaluating the Effectiveness of Coverage-Guided Fuzzing for Testing Deep Learning Library APIscited by 1

Built on by

Not cited yet.