mlsanityData to 7 Oct 2026

KernelBench: Can LLMs Write Efficient GPU Kernels?

arXiv 2502.10517 ↗First posted 14 Feb 2025cs.LG cs.AI cs.PFcited by 206

Not on the map. Why?

Cited by 73 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Efficient GPU kernels are crucial for building performant machine learning architectures, but writing them is a time-consuming challenge that requires significant expertise; therefore, we explore using language models (LMs) to automate kernel generation. We introduce KernelBench, an open-source framework for evaluating LMs' ability to write fast and correct kernels on a suite of 250 carefully selected PyTorch ML workloads. KernelBench represents a real-world engineering environment and making progress on the introduced benchmark directly translates to faster practical kernels. We introduce a new evaluation metric fast_p, which measures the percentage of generated kernels that are functionally correct and offer a speedup greater than an adjustable threshold p over baseline. Our experiments across various state-of-the-art models and test-time methods show that frontier reasoning models perform the best out of the box but still fall short overall, matching the PyTorch baseline in less than 20% of the cases. While we show that results can improve by leveraging execution and profiling feedback during iterative refinement, KernelBench remains a challenging benchmark, with its difficulty increasing as we raise speedup threshold p.

Key facts. Cited by 206 papers in the map (data through 7 Oct 2026). Builds on 25 of its 48 references that are in the map. First posted on arXiv 14 Feb 2025.

Builds on

25 of 48 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  5. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  6. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  7. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  8. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  9. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  10. 2020Efficient Transformers: A Surveycited by 784
  11. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  12. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
13 more
  1. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  2. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 198
  3. 2024Simple linear attention language models balance the recall-throughput tradeoffcited by 197
  4. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 196
  5. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 153
  6. 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 86
  7. 2024Can Language Models Solve Olympiad Programming?cited by 70
  8. 2024ThunderKittens: Simple, Fast, and Adorable AI Kernelscited by 34
  9. 2023Comparing Llama-2 and GPT-3 LLMs for HPC kernels generationcited by 30
  10. 2024ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness?cited by 25
  11. 2024Performance-Aligned LLMs for Generating Fast Codecited by 20
  12. 2024CodeRosetta: Pushing the Boundaries of Unsupervised Code Translation for Parallel Programmingcited by 19
  13. 2024Priority Sampling of Large Language Models for Compilerscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 45; 2026: 161.

206 papers.

Sort:

By area:

  1. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 94
  2. 2025CUDA-L1: Improving CUDA Optimization via Contrastive Reinforcement Learningcited by 59
  3. 2025Towards Robust Agentic CUDA Kernel Benchmarking, Verification, and Optimizationcited by 52
  4. 2025Barbarians at the Gate: How AI is Upending Systems Researchcited by 48
  5. 2026ProgramBench: Can Language Models Rebuild Programs From Scratch?cited by 47
  6. 2025Astra: A Multi-Agent System for GPU Kernel Performance Optimizationcited by 46
  7. 2025CudaForge: An Agent Framework with Hardware Feedback for CUDA Kernel Optimizationcited by 42
  8. 2026CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generationcited by 41
  9. 2025AutoTriton: Automatic Triton Programming with Reinforcement Learning in LLMscited by 38
  10. 2026CORAL: Towards Autonomous Multi-Agent Evolution for Open-Ended Discoverycited by 38
  11. 2025CUDA-LLM: LLMs Can Write Efficient CUDA Kernelscited by 33
  12. 2026SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agentscited by 25
38 more
  1. 2025AlgoTune: Can Language Models Speed Up General-Purpose Numerical Programs?cited by 24
  2. 2025TritonRL: Training LLMs to Think and Code Triton Without Cheatingcited by 24
  3. 2025STARK: Strategic Team of Agents for Refining Kernelscited by 23
  4. 2025GPU Kernel Scientist: An LLM-Driven Framework for Iterative Kernel Optimizationcited by 22
  5. 2025AccelOpt: A Self-Improving LLM Agentic System for AI Accelerator Kernel Optimizationcited by 21
  6. 2026Dr. Kernel: Reinforcement Learning Done Right for Triton Kernel Generationscited by 21
  7. 2026FlashInfer-Bench: Building the Virtuous Cycle for AI-driven LLM Systemscited by 20
  8. 2025FrontierCS: Evolving Challenges for Evolving Intelligencecited by 19
  9. 2026K-Search: LLM Kernel Generation via Co-Evolving Intrinsic World Modelcited by 17
  10. 2025QiMeng-Kernel: Macro-Thinking Micro-Coding Paradigm for LLM-Based High-Performance GPU Kernel Generationcited by 16
  11. 2025SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?cited by 15
  12. 2026ResearchGym: Evaluating Language Model Agents on Real-World AI Researchcited by 15
  13. 2026AutoKernel: Autonomous GPU Kernel Optimization via Iterative Agent-Driven Searchcited by 14
  14. 2025EvoEngineer: Mastering Automated CUDA Kernel Code Evolution with Large Language Modelscited by 13
  15. 2025ConCuR: Conciseness Makes State-of-the-Art Kernel Generationcited by 13
  16. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  17. 2025CUDA-L2: Surpassing cuBLAS Performance for Matrix Multiplication through Reinforcement Learningcited by 13
  18. 2025Training Versatile Coding Agents in Synthetic Environmentscited by 13
  19. 2026SOL-ExecBench: Speed-of-Light Benchmarking for Real-World GPU Kernels Against Hardware Limitscited by 13
  20. 2026Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJackcited by 13
  21. 2025Autocomp: A Powerful and Portable Code Optimizer for Tensor Acceleratorscited by 12
  22. 2025SwizzlePerf: Hardware-Aware LLMs for GPU Kernel Performance Optimizationcited by 11
  23. 2025PRAGMA: A Profiling-Reasoned Multi-Agent Framework for Automatic Kernel Optimizationcited by 11
  24. 2026KernelFoundry: Hardware-aware evolutionary GPU kernel optimizationcited by 11
  25. 2026AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?cited by 11
  26. 2026KernelBlaster: Continual Cross-Task CUDA Optimization via Memory-Augmented In-Context Reinforcement Learningcited by 10
  27. 2026CUDABench: Benchmarking LLMs for Text-to-CUDA Generationcited by 10
  28. 2026KernelBenchX: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernelscited by 10
  29. 2026Automated Benchmark Auditing for AI Agents and Large Language Modelscited by 10
  30. 2026AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvementcited by 10
  31. 2026Towards Automated Kernel Generation in the Era of LLMscited by 9
  32. 2026KernelSkill: A Multi-Agent Framework for GPU Kernel Optimizationcited by 9
  33. 2026Dream-RSI: Recursive Self-Improvement through Evolving Worldscited by 9
  34. 2025LIFT: LLM-Based Pragma Insertion for HLS via GNN Supervised Fine-Tuningcited by 8
  35. 2025ProofWright: Towards Agentic Formal Verification of CUDAcited by 8
  36. 2026Understanding the Challenges in Iterative Generative Optimization with LLMscited by 8
  37. 2026Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loopscited by 8
  38. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8

50 most cited of 206 All 206 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026KernelZero: Co-Evolving Proposer and Coder for Continuously Improved GPU Kernel Generation
  2. 2026KernelSight-LM: A Kernel-Level LLM Inference Simulator
  3. 2026GPU Forecasters: Language Models as Selective Surrogates for Kernel Runtime Optimization
  4. 2025Can Large Language Models Predict Parallel Code Performance?
  5. 2026AscendKernelGen: A Systematic Study of LLM-Based Kernel Generation for Neural Processing Units
  6. 2026D2K-Bench: Can LLM Agents Turn Expert Designs into Efficient GPU Kernels?
  7. 2023Efficient LLM inference solution on Intel GPU
  8. 2025Breaking the Boundaries of Long-Context LLM Inference: Adaptive KV Management on a Single Commodity GPU