# Geak: Introducing Triton Kernel AI Agent & Evaluation Benchmarks

arXiv: https://arxiv.org/abs/2507.23194 · HTML: https://mlsanity.com/paper/2507.23194/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 9 references that are in the map.
- First posted on arXiv 31 Jul 2025.

## Builds on (most cited first, 6 of 6)

- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [TritonBench: Benchmarking Large Language Model Capabilities for Generating Triton Operators](https://mlsanity.com/paper/2502.14752/) (2025)
- [EffiBench: Benchmarking the Efficiency of Automatically Generated Code](https://mlsanity.com/paper/2402.02037/) (2024)
- [How Efficient is LLM-Generated Code? A Rigorous & High-Standard Benchmark](https://mlsanity.com/paper/2406.06647/) (2024)
- [Can Large Language Models Write Parallel Code?](https://mlsanity.com/paper/2401.12554/) (2024)

## Built on by (most cited first, 30 of 32)

- [Characterizing Deep Research: A Benchmark and Formal Definition](https://mlsanity.com/paper/2508.04183/) (2025)
- [AutoKernel: Autonomous GPU Kernel Optimization via Iterative Agent-Driven Search](https://mlsanity.com/paper/2603.21331/) (2026)
- [EvoEngineer: Mastering Automated CUDA Kernel Code Evolution with Large Language Models](https://mlsanity.com/paper/2510.03760/) (2025)
- [KernelFoundry: Hardware-aware evolutionary GPU kernel optimization](https://mlsanity.com/paper/2603.12440/) (2026)
- [KernelBenchX: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernels](https://mlsanity.com/paper/2605.04956/) (2026)
- [Towards Automated Kernel Generation in the Era of LLMs](https://mlsanity.com/paper/2601.15727/) (2026)
- [KernelSkill: A Multi-Agent Framework for GPU Kernel Optimization](https://mlsanity.com/paper/2603.10085/) (2026)
- [AKG kernel Agent: A Multi-Agent Framework for Cross-Platform Kernel Synthesis](https://mlsanity.com/paper/2512.23424/) (2025)
- [The Correctness Illusion in LLM-Generated GPU Kernels](https://mlsanity.com/paper/2606.20128/) (2026)
- [KernelEvolve: Scaling Agentic Kernel Coding for Heterogeneous AI Accelerators at Meta](https://mlsanity.com/paper/2512.23236/) (2025)
- [KernelCraft: Benchmarking for Agentic Close-to-Metal Kernel Generation on Emerging Hardware](https://mlsanity.com/paper/2603.08721/) (2026)
- [A Two-Stage GPU Kernel Tuner Combining Semantic Refactoring and Search-Based Optimization](https://mlsanity.com/paper/2601.12698/) (2026)
- [OptiML: An End-to-End Framework for Program Synthesis and CUDA Kernel Optimization](https://mlsanity.com/paper/2602.12305/) (2026)
- [DRTriton: Large-Scale Synthetic Data Driven Reinforcement Learning for Triton Kernel Generation](https://mlsanity.com/paper/2603.21465/) (2026)
- [AscendOptimizer: Episodic Agent for Ascend NPU Operator Optimization](https://mlsanity.com/paper/2603.23566/) (2026)
- [KernelBand: Steering LLM-based Kernel Optimization via Hardware-Aware Multi-Armed Bandits](https://mlsanity.com/paper/2511.18868/) (2025)
- [AgentKernelArena: Generalization-Aware Benchmarking of GPU Kernel Optimization Agents](https://mlsanity.com/paper/2605.16819/) (2026)
- [Xe-Forge: Multi-Stage LLM-Powered Kernel Optimization for Intel GPU](https://mlsanity.com/paper/2605.26118/) (2026)
- [From Human Guidance to Autonomy: Agent Skill System for End-to-End LLM Deployment on Spatial NPUs](https://mlsanity.com/paper/2606.07586/) (2026)
- [Operator-Aware Mixed-Precision Tolerance Calibration for Tensor Kernels](https://mlsanity.com/paper/2607.16228/) (2026)
- [CANN Bench: Benchmarking Agent Generated Kernels against Real NPU and Algorithmic Limits](https://mlsanity.com/paper/2607.20518/) (2026)
- [Kernel Forge: An Agent Harness for LLM-based Generation and Optimization of CUDA Kernels](https://mlsanity.com/paper/2607.24762/) (2026)
- [EGG: An Expert-Guided Agent Framework for Kernel Generation](https://mlsanity.com/paper/2606.26758/) (2026)
- [Test-Input Generation for Tensor Programs: What Actually Finds Kernel Bugs](https://mlsanity.com/paper/2606.27396/) (2026)
- [KernelGenBench: Can LLMs and Agents Write Efficient Kernels Across Operator Sources and Hardware Platforms?](https://mlsanity.com/paper/2607.27231/) (2026)
- [AtumAI: A Principled Framework for Agentic Generation of Datacenter Control-Plane Policies](https://mlsanity.com/paper/2608.02569/) (2026)
- [PTXBench: Benchmarking and Adapting LLMs for GPU Kernel Optimization with Architecture-specific PTX](https://mlsanity.com/paper/2608.17379/) (2026)
- [AsmEvo: Agentic Assembly-Level Optimization of AMD GPU Kernels with Functional Equivalence Verification](https://mlsanity.com/paper/2608.20711/) (2026)
- [HIERA: Workload-Aware Planning Across Implementation Spaces for GPU Kernel Optimization](https://mlsanity.com/paper/2608.21157/) (2026)
- [AMDKernelVault: Large-Scale Datasets and Agentic Training for AMD GPU Kernel Optimization](https://mlsanity.com/paper/2609.12471/) (2026)

## Abstract (arXiv)

The demand for AI-generated GPU kernels is rapidly growing, influenced by the need for scalable, hardware-optimized solutions in both industry and academia. As deep learning workloads grow in complexity and diversity, it is imperative to automate low-level kernel development to meet performance and productivity demands. Major cloud providers, semiconductor companies, and research institutions are now investing heavily in AI-driven code generation for GPUs, aiming to reduce manual optimization efforts while achieving near-expert performance on hardware like AMD MI300X. The Triton language, a Python-based DSL for GPU programming, has emerged as a popular target for such AI-generated kernels due to its balance of performance and ease-of-coding. In this work, we present an evaluation suite for Triton-based GPU kernels and GEAK (Generating Efficient AI-centric GPU Kernels)-a framework that leverages cutting-edge LLMs to generate performant Triton code specifically for AMD GPUs, including the AMD MI300X and MI250. GEAK leverages inference-time compute scaling to produce Triton-based GPU kernels using a reasoning loop adapted from Reflexion-style feedback mechanisms. On two evaluation benchmarks, GEAK significantly outperformed the baselines of directly prompting frontier LLMs as well as Reflexion-based generation pipelines by achieving correctness up to 63% and execution speed up of up to 2.59X. These results highlight the promise of GEAK-like agentic code generation for accelerating the adoption of diverse hardware platforms and democratizing access to expert-level kernel performance.