# KGym: A Platform and Dataset to Benchmark Large Language Models on Linux Kernel Crash Resolution

arXiv: https://arxiv.org/abs/2407.02680 · HTML: https://mlsanity.com/paper/2407.02680/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jul 2024.

## Built on by (most cited first, 12 of 12)

- [KNighter: Transforming Static Analysis with LLM-Synthesized Checkers](https://mlsanity.com/paper/2503.09002/) (2025)
- [Code Researcher: Deep Research Agent for Large Systems Code and Commit History](https://mlsanity.com/paper/2506.11060/) (2025)
- [REFINE: Enhancing Program Repair Agents through Context-Aware Patch Refinement](https://mlsanity.com/paper/2510.03588/) (2025)
- [Stabilizing Long-term Multi-turn Reinforcement Learning with Gated Rewards](https://mlsanity.com/paper/2508.10548/) (2025)
- [kAgent: An execution-guided crash resolution agent for the Linux kernel](https://mlsanity.com/paper/2504.20412/) (2025)
- [AppForge: From Assistant to Independent Developer -- Are GPTs Ready for Software Development?](https://mlsanity.com/paper/2510.07740/) (2025)
- [Agentic Harness for Real-World Compilers](https://mlsanity.com/paper/2603.20075/) (2026)
- [Taming System Complexity: Demystifying Software Engineering Agents in Diagnosing Linux Kernel Faults](https://mlsanity.com/paper/2505.19489/) (2025)
- [LLM-Driven Kernel Evolution: Automating Driver Updates in Linux](https://mlsanity.com/paper/2511.18924/) (2025)
- [Learning From Developers: Towards Reliable Patch Validation at Scale for Linux](https://mlsanity.com/paper/2603.24825/) (2026)
- [LLVM-Bench: Benchmarking and Advancing Large Language Models for LLVM Compiler Issue Resolution](https://mlsanity.com/paper/2607.00700/) (2026)
- [TRIM: Reducing AI-Generated CodeSlop via Agent Trajectory Minimization](https://mlsanity.com/paper/2607.18161/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are consistently improving at increasingly realistic software engineering (SE) tasks. In real-world software stacks, significant SE effort is spent developing foundational system software like the Linux kernel. Unlike application-level software, a systems codebase like Linux is multilingual (low-level C/Assembly/Bash/Rust); gigantic (>20 million lines); critical (impacting billions of devices worldwide), and highly concurrent (involving complex multi-threading). To evaluate if ML models are useful while developing such large-scale systems-level software, we introduce kGym (a platform) and kBench (a dataset). The kGym platform provides a SE environment for large-scale experiments on the Linux kernel, including compiling and running kernels in parallel across several virtual machines, detecting operations and crashes, inspecting logs, and querying and patching the code base. We use kGym to facilitate evaluation on kBench, a crash resolution benchmark drawn from real-world Linux kernel bugs. An example bug in kBench contains crashing stack traces, a bug-reproducer file, a developer-written fix, and other associated data. To understand current performance, we conduct baseline experiments by prompting LLMs to resolve Linux kernel crashes. Our initial evaluations reveal that the best performing LLM achieves 0.72% and 5.38% in the unassisted and assisted (i.e., buggy files disclosed to the model) settings, respectively. These results highlight the need for further research to enhance model performance in SE tasks. Improving performance on kBench requires models to master new learning skills, including understanding the cause of crashes and repairing faults, writing memory-safe and hardware-aware code, and understanding concurrency. As a result, this work opens up multiple avenues of research at the intersection of machine learning and systems software.