Counting Without Running: Evaluating LLMs' Reasoning About Code Complexity
On the map: LLM reasoning → Reasoning for agents, planning and code
Abstract (arXiv)
Modern GPU software stacks demand developers who can anticipate performance bottlenecks before ever launching a kernel; misjudging floating-point workloads upstream can derail tuning, scheduling, and even hardware procurement. Yet despite rapid progress in code generation, today's Large Language Models (LLMs) are rarely tested on this kind of forward-looking reasoning. We close that gap with gpuFLOPBench, a benchmark that asks models to "count without running" by predicting single and double-precision FLOP counts for 577 CUDA kernels drawn from HeCBench, annotated with ground-truth profiles and eight execution attributes that distinguish trivially analyzable code from kernels whose FLOPs depend on hidden compiler or runtime behavior. Evaluating current closed-source reasoning models shows clear but uneven progress: the newest LLMs achieve perfect classification on straightforward kernels but still incur multiple order-of-magnitude errors whenever implicit FLOPs arise from division, intrinsic math functions, or common subexpressions. These results surface a core limitation of existing code assistants -- the inability to internalize hardware-specific microcode effects -- and position gpuFLOPBench as a focused testbed for developing LLM tooling that can reason about performance with the same rigor as experienced GPU developers. Sources are available at our repository: https://github.com/Scientific-Computing-Lab/gpuFLOPBench
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 15 of its 27 references that are in the map. First posted on arXiv 4 Dec 2025.
Builds on
15 of 27 references.
Closest ancestors · 4
Same direction, newest first
- 2025Towards Robust Agentic CUDA Kernel Benchmarking, Verification, and Optimization16 Sep
- 2025CoRe: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Tasks3 Jul
- 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code12 Mar
- 2022Competition-Level Code Generation with AlphaCode8 Feb
Other references · 11
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2021Program Synthesis with Large Language Modelscited by 3,509
- 2021Measuring Coding Challenge Competence With APPScited by 879
- 2025KernelBench: Can LLMs Write Efficient GPU Kernels?cited by 206
- 2025TritonBench: Benchmarking Large Language Model Capabilities for Generating Triton Operatorscited by 68
- 2023Large Language Models for Code Analysis: Do LLMs Really Do Their Job?cited by 44
- 2018Ithemal: Accurate, Portable and Fast Basic Block Throughput Estimation using Deep Neural Networkscited by 43
- 2025MultiKernelBench: A Multi-Platform Benchmark for Kernel Generationcited by 35
- 2024The Landscape and Challenges of HPC Research and LLMscited by 22
- 2025Can Large Language Models Predict Parallel Code Performance?cited by 4
- 2025Omniwise: Predicting GPU Kernels Performance with LLMscited by 1
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026KernelBench-Verified: Do LLM-Generated Kernels Actually Beat PyTorch?
- 2025Integrating Performance Tools in Model Reasoning for GPU Kernel Optimization
- 2026FastKernels: Benchmarking GPU Kernel Generation in Production
- 2026PerfReasoning: How Well Do LLMs Reason on Hardware Performance?
- 2026Optimizing CUDA like a Human: Micro-Profiling Tools as Expert Surrogates for LLM-Based GPU Kernel Optimization
- 2026Are LLM-Generated GPU Kernels Production-Ready? A Trace-Driven Benchmark and Optimization Agent
- 2026CoRE: A Fine-Grained Code Reasoning Benchmark Beyond Output Prediction
- 2025ProofWright: Towards Agentic Formal Verification of CUDA