# SOL-ExecBench: Speed-of-Light Benchmarking for Real-World GPU Kernels Against Hardware Limits arXiv: https://arxiv.org/abs/2603.19173 ยท HTML: https://mlsanity.com/paper/2603.19173/ ## Key facts - Cited by 18 papers in the map (data through 14 Oct 2026). - Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. - Builds on 14 of its 26 references that are in the map. - First posted on arXiv 19 Mar 2026. ## Placed in - LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/) ## Builds on (most cited first, 14 of 14) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021) - [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022) - [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024) - [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023) - [Gated Delta Networks: Improving Mamba2 with Delta Rule](https://mlsanity.com/paper/2412.06464/) (2024) - [StarCoder 2 and The Stack v2: The Next Generation](https://mlsanity.com/paper/2402.19173/) (2024) - [KernelBench: Can LLMs Write Efficient GPU Kernels?](https://mlsanity.com/paper/2502.10517/) (2025) - [TritonBench: Benchmarking Large Language Model Capabilities for Generating Triton Operators](https://mlsanity.com/paper/2502.14752/) (2025) - [Towards Robust Agentic CUDA Kernel Benchmarking, Verification, and Optimization](https://mlsanity.com/paper/2509.14279/) (2025) - [FlashInfer-Bench: Building the Virtuous Cycle for AI-driven LLM Systems](https://mlsanity.com/paper/2601.00227/) (2026) - [CUDABench: Benchmarking LLMs for Text-to-CUDA Generation](https://mlsanity.com/paper/2603.02236/) (2026) - [VibeTensor: System Software for Deep Learning, Fully Generated by AI Agents](https://mlsanity.com/paper/2601.16238/) (2026) - [The EDGE Language: Extended General Einsums for Graph Algorithms](https://mlsanity.com/paper/2404.11591/) (2024) ## Built on by (most cited first, 18 of 18) - [Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning](https://mlsanity.com/paper/2606.15007/) (2026) - [The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement](https://mlsanity.com/paper/2609.11873/) (2026) - [FastKernels: Benchmarking GPU Kernel Generation in Production](https://mlsanity.com/paper/2605.23215/) (2026) - [Are LLM-Generated GPU Kernels Production-Ready? A Trace-Driven Benchmark and Optimization Agent](https://mlsanity.com/paper/2607.14541/) (2026) - [Xe-Forge: Multi-Stage LLM-Powered Kernel Optimization for Intel GPU](https://mlsanity.com/paper/2605.26118/) (2026) - [CANN Bench: Benchmarking Agent Generated Kernels against Real NPU and Algorithmic Limits](https://mlsanity.com/paper/2607.20518/) (2026) - [Argus: A General-Purpose Agentic Reasoning Runtime for Long-Horizon Tasks](https://mlsanity.com/paper/2608.05144/) (2026) - [Stateless Language Agents: Scaling Long-Horizon Automated Research](https://mlsanity.com/paper/2610.07625/) (2026) - [KForge: LLM-Driven Cross-Platform Kernel Generation for AI Accelerators](https://mlsanity.com/paper/2606.02963/) (2026) - [SOLAR: AI-Powered Speed-of-Light Performance Analysis](https://mlsanity.com/paper/2606.26383/) (2026) - [KernelGenBench: Can LLMs and Agents Write Efficient Kernels Across Operator Sources and Hardware Platforms?](https://mlsanity.com/paper/2607.27231/) (2026) - [PTXBench: Benchmarking and Adapting LLMs for GPU Kernel Optimization with Architecture-specific PTX](https://mlsanity.com/paper/2608.17379/) (2026) - [PerfReasoning: How Well Do LLMs Reason on Hardware Performance?](https://mlsanity.com/paper/2609.04476/) (2026) - [SWE-Serve: Benchmarking Agentic Engineering For Production Inference Serving](https://mlsanity.com/paper/2609.26777/) (2026) - [D2K-Bench: Can LLM Agents Turn Expert Designs into Efficient GPU Kernels?](https://mlsanity.com/paper/2610.03226/) (2026) - [MetaKernelBench: Measuring GPU Kernel Knowledge Transfer Beyond Code](https://mlsanity.com/paper/2610.05014/) (2026) - [Cleave: Scaling Tensor Program Optimization via Decoupled Algebraic Search and Operator Scheduling](https://mlsanity.com/paper/2610.07742/) (2026) - [DuoDiscover: Scaling Autonomous Discovery Beyond the Evaluation Bottleneck](https://mlsanity.com/paper/2610.17184/) (2026) ## Abstract (arXiv) As agentic AI systems become increasingly capable of generating and optimizing GPU kernels, progress is constrained by benchmarks that reward speedup over software baselines rather than proximity to hardware-efficient execution. We present SOL-ExecBench, a benchmark of 235 CUDA kernel optimization problems extracted from 124 production and emerging AI models spanning language, diffusion, vision, audio, video, and hybrid architectures, targeting NVIDIA Blackwell GPUs. The benchmark covers forward and backward workloads across BF16, FP8, and NVFP4, including kernels whose best performance is expected to rely on Blackwell-specific capabilities. Unlike prior benchmarks that evaluate kernels primarily relative to software implementations, SOL-ExecBench measures performance against analytically derived Speed-of-Light (SOL) bounds computed by SOLAR, our pipeline for deriving hardware-grounded SOL bounds, yielding a fixed target for hardware-efficient optimization. We report a SOL Score that quantifies how much of the gap between a release-defined scoring baseline and the hardware SOL bound a candidate kernel closes. To support robust evaluation of agentic optimizers, we additionally provide a sandboxed harness with GPU clock locking, L2 cache clearing, isolated subprocess execution, and static analysis based checks against common reward-hacking strategies. SOL-ExecBench reframes GPU kernel benchmarking from beating a mutable software baseline to closing the remaining gap to hardware Speed-of-Light.