mlsanityData to 7 Oct 2026

TritonBench: Benchmarking Large Language Model Capabilities for Generating Triton Operators

arXiv 2502.14752 ↗First posted 20 Feb 2025cs.CL cs.LGcited by 68

Not on the map. Why?

Abstract (arXiv)

Triton, a high-level Python-like language designed for building efficient GPU kernels, is widely adopted in deep learning frameworks due to its portability, flexibility, and accessibility. However, programming and parallel optimization still require considerable trial and error from Triton developers. Despite advances in large language models (LLMs) for conventional code generation, these models struggle to generate accurate, performance-optimized Triton code, as they lack awareness of its specifications and the complexities of GPU programming. More critically, there is an urgent need for systematic evaluations tailored to Triton. In this work, we introduce TritonBench, the first comprehensive benchmark for Triton operator generation. TritonBench features two evaluation channels: a curated set of 184 real-world operators from GitHub and a collection of operators aligned with PyTorch interfaces. Unlike conventional code benchmarks prioritizing functional correctness, TritonBench also profiles efficiency performance on widely deployed GPUs aligned with industry applications. Our study reveals that current state-of-the-art code LLMs struggle to generate efficient Triton operators, highlighting a significant gap in high-performance code generation. TritonBench will be available at https://github.com/thunlp/TritonBench.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Feb 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 14; 2026: 54.

68 papers.

Sort:

By area:

  1. 2025Kevin: Multi-Turn RL for Generating CUDA Kernelscited by 73
  2. 2025Astra: A Multi-Agent System for GPU Kernel Performance Optimizationcited by 46
  3. 2025AutoTriton: Automatic Triton Programming with Reinforcement Learning in LLMscited by 38
  4. 2025Geak: Introducing Triton Kernel AI Agent & Evaluation Benchmarkscited by 31
  5. 2025TritonRL: Training LLMs to Think and Code Triton Without Cheatingcited by 24
  6. 2026FlashInfer-Bench: Building the Virtuous Cycle for AI-driven LLM Systemscited by 20
  7. 2025QiMeng-Kernel: Macro-Thinking Micro-Coding Paradigm for LLM-Based High-Performance GPU Kernel Generationcited by 16
  8. 2025ConCuR: Conciseness Makes State-of-the-Art Kernel Generationcited by 13
  9. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  10. 2026SOL-ExecBench: Speed-of-Light Benchmarking for Real-World GPU Kernels Against Hardware Limitscited by 13
  11. 2026KernelFoundry: Hardware-aware evolutionary GPU kernel optimizationcited by 11
  12. 2026CUDABench: Benchmarking LLMs for Text-to-CUDA Generationcited by 10
38 more
  1. 2026KernelBenchX: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernelscited by 10
  2. 2026Towards Automated Kernel Generation in the Era of LLMscited by 9
  3. 2025AKG kernel Agent: A Multi-Agent Framework for Cross-Platform Kernel Synthesiscited by 7
  4. 2026StitchCUDA: An Automated Multi-Agents End-to-End GPU Programing Framework with Rubric-based Agentic Reinforcement Learningcited by 7
  5. 2026ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?cited by 6
  6. 2026KernelCraft: Benchmarking for Agentic Close-to-Metal Kernel Generation on Emerging Hardwarecited by 4
  7. 2026InCoder-32B-Thinking: Industrial Code World Model for Thinkingcited by 4
  8. 2025GPU Kernel Optimization Beyond Full Builds: An LLM Framework with Minimal Executable Programscited by 3
  9. 2026A Two-Stage GPU Kernel Tuner Combining Semantic Refactoring and Search-Based Optimizationcited by 3
  10. 2026OptiML: An End-to-End Framework for Program Synthesis and CUDA Kernel Optimizationcited by 3
  11. 2026Making LLMs Optimize Multi-Scenario CUDA Kernels Like Expertscited by 3
  12. 2026Towards Cold-Start Drafting and Continual Refining: A Value-Driven Memory Approach with Application to NPU Kernel Synthesiscited by 3
  13. 2026AdaExplore: Failure-Driven Adaptation and Diversity-Preserving Search for Efficient Kernel Generationcited by 3
  14. 2025KernelBand: Steering LLM-based Kernel Optimization via Hardware-Aware Multi-Armed Banditscited by 2
  15. 2025Counting Without Running: Evaluating LLMs' Reasoning About Code Complexitycited by 2
  16. 2026AscendOptimizer: Episodic Agent for Ascend NPU Operator Optimizationcited by 2
  17. 2026FastKernels: Benchmarking GPU Kernel Generation in Productioncited by 2
  18. 2026daVinci-kernel: Co-Evolving Skill Selection, Summarization, and Utilization via RL for GPU Kernel Optimizationcited by 2
  19. 2026NKI-Agent: Domain-Specific Fine-Tuning and Agentic Tool Use for Neuron Kernel Generationcited by 2
  20. 2026Are LLM-Generated GPU Kernels Production-Ready? A Trace-Driven Benchmark and Optimization Agentcited by 2
  21. 2026PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimizationcited by 2
  22. 2025Evolution of Kernels: Automated RISC-V Kernel Optimization with Large Language Modelscited by 1
  23. 2026DiffBench Meets DiffAgent: End-to-End LLM-Driven Diffusion Acceleration Code Generationcited by 1
  24. 2026DICE: Diffusion Large Language Models Excel at Generating CUDA Kernelscited by 1
  25. 2026DRTriton: Large-Scale Synthetic Data Driven Reinforcement Learning for Triton Kernel Generationcited by 1
  26. 2026CuBridge: An LLM-Based Framework for Understanding and Reconstructing High-Performance Attention Kernelscited by 1
  27. 2026CUDAHercules: Benchmarking Hardware-Aware Expert-level CUDA Optimization for LLMscited by 1
  28. 2026Metal-Sci: A Scientific Compute Benchmark for Evolutionary LLM Kernel Search on Apple Siliconcited by 1
  29. 2026Xe-Forge: Multi-Stage LLM-Powered Kernel Optimization for Intel GPUcited by 1
  30. 2026JAXBench: Benchmarking Autonomous TPU Kernel Optimizationcited by 1
  31. 2026CANN Bench: Benchmarking Agent Generated Kernels against Real NPU and Algorithmic Limitscited by 1
  32. 2026Kernel Forge: An Agent Harness for LLM-based Generation and Optimization of CUDA Kernelscited by 1
  33. 2026AgenticCANN: Automated Ascend C Operator Generation via Knowledge-Augmented Agentic Evolutioncited by 1
  34. 2026RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI Frameworkscited by 1
  35. 2025Memory-Efficient Acceleration of Block Low-Rank Foundation Models on Resource Constrained GPUscited by 0
  36. 2026The New Compiler Stack: A Survey on the Synergy of LLMs and Compilerscited by 0
  37. 2026MobileKernelBench: Can LLMs Write Efficient Kernels for Mobile Devices?cited by 0
  38. 2026CUDABeaver: Benchmarking LLM-Based Automated CUDA Debuggingcited by 0

50 most cited of 68 All 68 (JSON)