# Agentic Harness for Real-World Compilers

arXiv: https://arxiv.org/abs/2603.20075 · HTML: https://mlsanity.com/paper/2603.20075/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 18 of its 47 references that are in the map.
- First posted on arXiv 20 Mar 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 18 of 18)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [KernelBench: Can LLMs Write Efficient GPU Kernels?](https://mlsanity.com/paper/2502.10517/) (2025)
- [Fuzz4All: Universal Fuzzing with Large Language Models](https://mlsanity.com/paper/2308.04748/) (2023)
- [LocAgent: Graph-Guided LLM Agents for Code Localization](https://mlsanity.com/paper/2503.09089/) (2025)
- [CompilerGym: Robust, Performant Compiler Optimization Environments for AI Research](https://mlsanity.com/paper/2109.08267/) (2021)
- [A Survey on Compiler Autotuning using Machine Learning](https://mlsanity.com/paper/1801.04405/) (2018)
- [MLGO: a Machine Learning Guided Compiler Optimizations Framework](https://mlsanity.com/paper/2101.04808/) (2021)
- [WhiteFox: White-Box Compiler Fuzzing Empowered by Large Language Models](https://mlsanity.com/paper/2310.15991/) (2023)
- [LLM-Vectorizer: LLM-based Verified Loop Vectorizer](https://mlsanity.com/paper/2406.04693/) (2024)
- [LLM4VV: Developing LLM-Driven Testsuite for Compiler Validation](https://mlsanity.com/paper/2310.04963/) (2023)
- [CompileAgent: Automated Real-World Repo-Level Compilation with Tool-Integrated LLM-based Agent System](https://mlsanity.com/paper/2505.04254/) (2025)
- [KGym: A Platform and Dataset to Benchmark Large Language Models on Linux Kernel Crash Resolution](https://mlsanity.com/paper/2407.02680/) (2024)
- [RL4ReAl: Reinforcement Learning for Register Allocation](https://mlsanity.com/paper/2204.02013/) (2022)
- [LPO: Discovering Missed Peephole Optimizations with Large Language Models](https://mlsanity.com/paper/2508.16125/) (2025)
- [Interleaving Large Language Models for Compiler Testing](https://mlsanity.com/paper/2508.18955/) (2025)
- [Rethinking Kernel Program Repair: Benchmarking and Enhancing LLMs with RGym](https://mlsanity.com/paper/2511.15757/) (2025)

## Built on by (most cited first, 3 of 3)

- [Understanding Agent-Based Patching of Compiler Missed Optimizations](https://mlsanity.com/paper/2607.02370/) (2026)
- [Can Coding Agents Implement Missed Compiler Optimizations? Evaluating LLM Agents on LLVM Peephole Optimizations](https://mlsanity.com/paper/2607.02684/) (2026)
- [Semantic-aware and Self-improving Program Reduction via Agentic Large Language Models](https://mlsanity.com/paper/2607.03766/) (2026)

## Abstract (arXiv)

Compilers are critical to modern computing, yet fixing compiler bugs is difficult. While recent large language model (LLM) advancements enable automated bug repair, compiler bugs pose unique challenges due to their complexity, deep cross-domain expertise requirements, and sparse, non-descriptive bug reports, necessitating compiler-specific harnesses. To bridge the gap, we introduce llvm-harness, the first harness designed to assist LLM agents in understanding and fixing compiler bugs. Our current focus is on the middle end of LLVM, one of the most widely used compiler infrastructures. Central to llvm-harness are agent-friendly LLVM tools, a benchmark llvm-bench of 334 reproducible LLVM middle-end bugs, and a tailored mini agent llvm-autofix-mini for fixing LLVM middle-end bugs automatically. We evaluate five frontier models and find that they exhibit a performance decline when tackling compiler bugs with the state-of-the-art agent. With llvm-harness' enhancement, their performance improves by 62%. Our specialized mini agent llvm-autofix-mini further outperforms the llvm-harness-enhanced state-of-the-art by 22%. This emphasizes the necessity for specialized harnesses like ours to assist LLMs in compiler engineering tasks. Despite promising results, our expert review also reveals several open challenges that remain when applying LLMs for compiler engineering tasks. GitHub: https://github.com/dtcxzyw/llvm-harness