# FlashInfer-Bench: Building the Virtuous Cycle for AI-driven LLM Systems

arXiv: https://arxiv.org/abs/2601.00227 · HTML: https://mlsanity.com/paper/2601.00227/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 23 references that are in the map.
- First posted on arXiv 1 Jan 2026.

## Builds on (most cited first, 10 of 10)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision](https://mlsanity.com/paper/2407.08608/) (2024)
- [FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving](https://mlsanity.com/paper/2501.01005/) (2025)
- [KernelBench: Can LLMs Write Efficient GPU Kernels?](https://mlsanity.com/paper/2502.10517/) (2025)
- [Ansor: Generating High-Performance Tensor Programs for Deep Learning](https://mlsanity.com/paper/2006.06762/) (2020)
- [Kevin: Multi-Turn RL for Generating CUDA Kernels](https://mlsanity.com/paper/2507.11948/) (2025)
- [TritonBench: Benchmarking Large Language Model Capabilities for Generating Triton Operators](https://mlsanity.com/paper/2502.14752/) (2025)

## Built on by (most cited first, 25 of 25)

- [K-Search: LLM Kernel Generation via Co-Evolving Intrinsic World Model](https://mlsanity.com/paper/2602.19128/) (2026)
- [SOL-ExecBench: Speed-of-Light Benchmarking for Real-World GPU Kernels Against Hardware Limits](https://mlsanity.com/paper/2603.19173/) (2026)
- [Towards Automated Kernel Generation in the Era of LLMs](https://mlsanity.com/paper/2601.15727/) (2026)
- [InferenceBench: A Benchmark for Open-Ended LLM Inference Optimization by AI Agents](https://mlsanity.com/paper/2607.20468/) (2026)
- [DICE: Diffusion Large Language Models Excel at Generating CUDA Kernels](https://mlsanity.com/paper/2602.11715/) (2026)
- [AdaExplore: Failure-Driven Adaptation and Diversity-Preserving Search for Efficient Kernel Generation](https://mlsanity.com/paper/2604.16625/) (2026)
- [FastKernels: Benchmarking GPU Kernel Generation in Production](https://mlsanity.com/paper/2605.23215/) (2026)
- [Are LLM-Generated GPU Kernels Production-Ready? A Trace-Driven Benchmark and Optimization Agent](https://mlsanity.com/paper/2607.14541/) (2026)
- [Harness Engineering for LLM-Driven GPU Kernel Generation](https://mlsanity.com/paper/2607.17979/) (2026)
- [JAXBench: Benchmarking Autonomous TPU Kernel Optimization](https://mlsanity.com/paper/2607.20466/) (2026)
- [RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI Frameworks](https://mlsanity.com/paper/2608.12004/) (2026)
- [When Agents Slow Down: Understanding LLM Agents' Test-Time Strategies via Elo-per-token Analysis](https://mlsanity.com/paper/2609.15309/) (2026)
- [MKEvolve: A Modular Multi-Agent Framework for Kernel Code Generation](https://mlsanity.com/paper/2607.20501/) (2026)
- [Kernel Forge: An Agent Harness for LLM-based Generation and Optimization of CUDA Kernels](https://mlsanity.com/paper/2607.24762/) (2026)
- [Towards Feedback-to-Plan Decisions for Self-Evolving LLM Agents in CUDA Kernel Generation](https://mlsanity.com/paper/2605.26720/) (2026)
- [PithTrain: A Compact and Agent-Native MoE Training System](https://mlsanity.com/paper/2605.31463/) (2026)
- [KernelGenBench: Can LLMs and Agents Write Efficient Kernels Across Operator Sources and Hardware Platforms?](https://mlsanity.com/paper/2607.27231/) (2026)
- [PTXBench: Benchmarking and Adapting LLMs for GPU Kernel Optimization with Architecture-specific PTX](https://mlsanity.com/paper/2608.17379/) (2026)
- [RoofLang: Enabling AI-Driven Architecting of LLM Inference Systems](https://mlsanity.com/paper/2609.12551/) (2026)
- [KernelZero: Co-Evolving Proposer and Coder for Continuously Improved GPU Kernel Generation](https://mlsanity.com/paper/2609.33074/) (2026)
- [EasyPPO: Stabilizing the Critic Is Key](https://mlsanity.com/paper/2609.36802/) (2026)
- [D2K-Bench: Can LLM Agents Turn Expert Designs into Efficient GPU Kernels?](https://mlsanity.com/paper/2610.03226/) (2026)
- [MetaKernelBench: Measuring GPU Kernel Knowledge Transfer Beyond Code](https://mlsanity.com/paper/2610.05014/) (2026)
- [AutoEnvScaling: Automating the Data Flywheel with Terminal Agents](https://mlsanity.com/paper/2610.14163/) (2026)
- [DuoDiscover: Scaling Autonomous Discovery Beyond the Evaluation Bottleneck](https://mlsanity.com/paper/2610.17184/) (2026)

## Abstract (arXiv)

Recent advances show that large language models (LLMs) can act as autonomous agents capable of generating GPU kernels, but integrating these AI-generated kernels into real-world inference systems remains challenging. FlashInfer-Bench addresses this gap by establishing a standardized, closed-loop framework that connects kernel generation, benchmarking, and deployment. At its core, FlashInfer Trace provides a unified schema describing kernel definitions, workloads, implementations, and evaluations, enabling consistent communication between agents and systems. Built on real serving traces, FlashInfer-Bench includes a curated dataset, a robust correctness- and performance-aware benchmarking framework, a public leaderboard to track LLM agents' GPU programming capabilities, and a dynamic substitution mechanism (apply()) that seamlessly injects the best-performing kernels into production LLM engines such as SGLang and vLLM. Using FlashInfer-Bench, we further evaluate the performance and limitations of LLM agents, compare the trade-offs among different GPU programming languages, and provide insights for future agent design. FlashInfer-Bench thus establishes a practical, reproducible pathway for continuously improving AI-generated kernels and deploying them into large-scale LLM inference.