# Evaluating and Improving Automated Repository-Level Rust Issue Resolution with LLM-based Agents

arXiv: https://arxiv.org/abs/2602.22764 · HTML: https://mlsanity.com/paper/2602.22764/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 32 of its 83 references that are in the map.
- First posted on arXiv 26 Feb 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 30 of 32)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [A Survey on the Memory Mechanism of Large Language Model based Agents](https://mlsanity.com/paper/2404.13501/) (2024)
- [Agentless: Demystifying LLM-based Software Engineering Agents](https://mlsanity.com/paper/2407.01489/) (2024)
- [RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation](https://mlsanity.com/paper/2303.12570/) (2023)
- [RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems](https://mlsanity.com/paper/2306.03091/) (2023)
- [Grounded Copilot: How Programmers Interact with Code-Generating Models](https://mlsanity.com/paper/2206.15000/) (2022)
- [Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving](https://mlsanity.com/paper/2504.02605/) (2025)
- [Multi-lingual Evaluation of Code Generation Models](https://mlsanity.com/paper/2210.14868/) (2022)
- [SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents](https://mlsanity.com/paper/2406.12952/) (2024)
- [SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents](https://mlsanity.com/paper/2504.08703/) (2025)
- [SpecRover: Code Intent Extraction via LLMs](https://mlsanity.com/paper/2408.02232/) (2024)
- [ComplexCodeEval: A Benchmark for Evaluating Large Code Models on More Complex Code](https://mlsanity.com/paper/2409.10280/) (2024)
- [Assessing and Advancing Benchmarks for Evaluating Large Language Models in Software Engineering Tasks](https://mlsanity.com/paper/2505.08903/) (2025)
- [How Far Can We Go with Practical Function-Level Program Repair?](https://mlsanity.com/paper/2404.12833/) (2024)
- [AEGIS: An Agent-based Framework for General Bug Reproduction from Issue Descriptions](https://mlsanity.com/paper/2411.18015/) (2024)
- [Syzygy: Dual Code-Test C to (safe) Rust Translation using LLMs and Dynamic Analysis](https://mlsanity.com/paper/2412.14234/) (2024)
- [Assessing the Quality of the Steps to Reproduce in Bug Reports](https://mlsanity.com/paper/1906.07107/) (2019)
- [A Study on Developer Behaviors for Validating and Repairing LLM-Generated Code Using Eye Tracking and IDE Actions](https://mlsanity.com/paper/2405.16081/) (2024)
- [CoderUJB: An Executable and Unified Java Benchmark for Practical Programming Scenarios](https://mlsanity.com/paper/2403.19287/) (2024)
- [Decompile-Bench: Million-Scale Binary-Source Function Pairs for Real-World Binary Decompilation](https://mlsanity.com/paper/2505.12668/) (2025)
- [RustEvo^2: An Evolving Benchmark for API Evolution in LLM-based Rust Code Generation](https://mlsanity.com/paper/2503.16922/) (2025)
- [Concrat: An Automatic C-to-Rust Lock API Translator for Concurrent Programs](https://mlsanity.com/paper/2301.10943/) (2023)
- [A Closer Look at the Security Risks in the Rust Ecosystem](https://mlsanity.com/paper/2308.15046/) (2023)
- [Grammar-Based Code Representation: Is It a Worthy Pursuit for LLMs?](https://mlsanity.com/paper/2503.05507/) (2025)
- [Rusty Linux: Advances in Rust for Linux Kernel Development](https://mlsanity.com/paper/2407.18431/) (2024)
- [Garbage Collection Makes Rust Easier to Use: A Randomized Controlled Trial of the Bronze Garbage Collector](https://mlsanity.com/paper/2110.01098/) (2021)

## Built on by (most cited first, 8 of 8)

- [SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks](https://mlsanity.com/paper/2603.24755/) (2026)
- [Empowering Autonomous Debugging Agents with Efficient Dynamic Analysis](https://mlsanity.com/paper/2604.24212/) (2026)
- [RustMizan: A Compilable, Contamination-Aware Benchmarking Framework for Rust Vulnerabilities](https://mlsanity.com/paper/2607.04729/) (2026)
- [MobileDev-Bench: A Benchmark for Issue Resolution in Mobile Application Development](https://mlsanity.com/paper/2603.24946/) (2026)
- [LLVM-Bench: Benchmarking and Advancing Large Language Models for LLVM Compiler Issue Resolution](https://mlsanity.com/paper/2607.00700/) (2026)
- [Generative Compilation: On-the-Fly Compiler Feedback as AI Generates Code](https://mlsanity.com/paper/2607.13921/) (2026)
- [SWE-Prometheus: Measuring Engineering Governance Improvements in Real-World Repositories](https://mlsanity.com/paper/2609.29465/) (2026)
- [SWE-MISBEHAVIOR: Benchmarking and Mitigating Boundary Violations in Software Coding Agents](https://mlsanity.com/paper/2610.18206/) (2026)

## Abstract (arXiv)

The Rust programming language presents a steep learning curve and significant coding challenges, making the automation of issue resolution essential for its broader adoption. Recently, LLM-powered code agents have shown remarkable success in resolving complex software engineering tasks, yet their application to Rust has been limited by the absence of a large-scale, repository-level benchmark. To bridge this gap, we introduce Rust-SWE-bench, a benchmark comprising 500 real-world, repository-level software engineering tasks from 34 diverse and popular Rust repositories. We then perform a comprehensive study on Rust-SWE-bench with four representative agents and four state-of-the-art LLMs to establish a foundational understanding of their capabilities and limitations in the Rust ecosystem. Our extensive study reveals that while ReAct-style agents are promising, i.e., resolving up to 21.2% of issues, they are limited by two primary challenges: comprehending repository-wide code structure and complying with Rust's strict type and trait semantics. We also find that issue reproduction is rather critical for task resolution. Inspired by these findings, we propose RUSTFORGER, a novel agentic approach that integrates an automated test environment setup with a Rust metaprogramming-driven dynamic tracing strategy to facilitate reliable issue reproduction and dynamic analysis. The evaluation shows that RUSTFORGER using Claude-Sonnet-3.7 significantly outperforms all baselines, resolving 28.6% of tasks on Rust-SWE-bench, i.e., a 34.9% improvement over the strongest baseline, and, in aggregate, uniquely solves 46 tasks that no other agent could solve across all adopted advanced LLMs.