# REDO: Execution-Free Runtime Error Detection for COding Agents

arXiv: https://arxiv.org/abs/2410.09117 · HTML: https://mlsanity.com/paper/2410.09117/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 21 of its 35 references that are in the map.
- First posted on arXiv 10 Oct 2024.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 21 of 21)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks](https://mlsanity.com/paper/2105.12655/) (2021)
- [Practical Program Repair in the Era of Large Pre-trained Language Models](https://mlsanity.com/paper/2210.14179/) (2022)
- [Learning Performance-Improving Code Edits](https://mlsanity.com/paper/2302.07867/) (2023)
- [IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities](https://mlsanity.com/paper/2405.17238/) (2024)
- [ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languages](https://mlsanity.com/paper/2212.06742/) (2022)
- [Detecting False Alarms from Automatic Static Analysis Tools: How Far are We?](https://mlsanity.com/paper/2202.05982/) (2022)
- [An Empirical Study of Static Analysis Tools for Secure Code Review](https://mlsanity.com/paper/2407.12241/) (2024)
- [Learning to Reduce False Positives in Analytic Bug Detectors](https://mlsanity.com/paper/2203.09907/) (2022)
- [Static Prediction of Runtime Errors by Learning to Execute Programs with External Resource Descriptions](https://mlsanity.com/paper/2203.03771/) (2022)
- [PyTy: Repairing Static Type Errors in Python](https://mlsanity.com/paper/2401.06619/) (2024)
- [An Extensive Comparison of Static Application Security Testing Tools](https://mlsanity.com/paper/2403.09219/) (2024)
- [Out of Context: How important is Local Context in Neural Program Repair?](https://mlsanity.com/paper/2312.04986/) (2023)
- [On (the Lack of) Code Confidentiality in Trusted Execution Environments](https://mlsanity.com/paper/2212.07899/) (2022)
- [An Empirical Study on Bug Severity Estimation using Source Code Metrics and Static Analysis](https://mlsanity.com/paper/2206.12927/) (2022)
- [FuzzSlice: Pruning False Positives in Static Analysis Warnings Through Function-Level Fuzzing](https://mlsanity.com/paper/2402.01923/) (2024)
- [A Static Analysis Platform for Investigating Security Trends in Repositories](https://mlsanity.com/paper/2304.01725/) (2023)

## Built on by (most cited first, 2 of 2)

- [Quasar: A Programming Language Specialized for LLM Code Actions](https://mlsanity.com/paper/2506.12202/) (2025)
- [Trustworthy Runtime Error Healing in Real-World Repositories: A Benchmark and Guardrail](https://mlsanity.com/paper/2609.39086/) (2026)

## Abstract (arXiv)

As LLM-based agents exhibit exceptional capabilities in addressing complex problems, there is a growing focus on developing coding agents to tackle increasingly sophisticated tasks. Despite their promising performance, these coding agents often produce programs or modifications that contain runtime errors, which can cause code failures and are difficult for static analysis tools to detect. Enhancing the ability of coding agents to statically identify such errors could significantly improve their overall performance. In this work, we introduce Execution-free Runtime Error Detection for COding Agents (REDO), a method that integrates LLMs with static analysis tools to detect runtime errors for coding agents, without code execution. Additionally, we propose a benchmark task, SWE-Bench-Error-Detection (SWEDE), based on SWE-Bench (lite), to evaluate error detection in repository-level problems with complex external dependencies. Finally, through both quantitative and qualitative analyses across various error detection tasks, we demonstrate that REDO outperforms current state-of-the-art methods by achieving a 11.0% higher accuracy and 9.1% higher weighted F1 score; and provide insights into the advantages of incorporating LLMs for error detection.