# Towards Effectively Leveraging Execution Traces for Program Repair with Code LLMs

arXiv: https://arxiv.org/abs/2505.04441 · HTML: https://mlsanity.com/paper/2505.04441/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 18 references that are in the map.
- First posted on arXiv 7 May 2025.

## Builds on (most cited first, 7 of 7)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://mlsanity.com/paper/2203.03850/) (2022)
- [CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks](https://mlsanity.com/paper/2105.12655/) (2021)
- [Embrace Divergence for Richer Insights: A Multi-document Summarization Benchmark and a Case Study on Summarizing Diverse Information from News Articles](https://mlsanity.com/paper/2309.09369/) (2023)
- [SelfAPR: Self-supervised Program Repair with Test Execution Diagnostics](https://mlsanity.com/paper/2203.12755/) (2022)
- [TRACED: Execution-aware Pre-training for Source Code](https://mlsanity.com/paper/2306.07487/) (2023)

## Built on by (most cited first, 11 of 11)

- [Automated Cloud Infrastructure-as-Code Reconciliation with AI Agents](https://mlsanity.com/paper/2510.20211/) (2025)
- [Debugging code world models](https://mlsanity.com/paper/2602.07672/) (2026)
- [HAFixAgent: History-Aware Program Repair Agent](https://mlsanity.com/paper/2511.01047/) (2025)
- [DynaFix: Iterative Automated Program Repair Driven by Execution-Level Dynamic Information](https://mlsanity.com/paper/2512.24635/) (2025)
- [Collaborative Agents for Automated Program Repair in Ruby](https://mlsanity.com/paper/2511.03925/) (2025)
- [ConVer: Using Contracts and Loop Invariant Synthesis for Scalable Formal Software Verification](https://mlsanity.com/paper/2605.27051/) (2026)
- [Multi-Perspective Agentic Program Repair via Code Property Graphs and Temporal Execution Graphs](https://mlsanity.com/paper/2607.12605/) (2026)
- [AuditRepairBench: A Paired-Execution Trace Corpus for Evaluator-Channel Ranking Instability in Agent Repair](https://mlsanity.com/paper/2605.04624/) (2026)
- [Detecting Architectural Drift in Safety-Critical Firmware through Runtime Trace Analysis](https://mlsanity.com/paper/2607.03135/) (2026)
- [How Do LLMs Read Bug Reports? An Empirical Study of Attention in LLMs for Automated Program Repair](https://mlsanity.com/paper/2607.25873/) (2026)
- [Evaluating RL Explainability Methods by How Much They Help Fix Bugs in Agents](https://mlsanity.com/paper/2608.17524/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) show promising performance on various programming tasks, including Automatic Program Repair (APR). However, most approaches to LLM-based APR are limited to the static analysis of the programs, while disregarding their runtime behavior. Inspired by knowledge-augmented NLP, in this work, we aim to remedy this potential blind spot by augmenting standard APR prompts with program execution traces. We evaluate our approach using the GPT family of models on three popular APR datasets. Our findings suggest that simply incorporating execution traces into the prompt provides a limited performance improvement over trace-free baselines, in only 2 out of 6 tested dataset / model configurations. We further find that the effectiveness of execution traces for APR diminishes as their complexity increases. We explore several strategies for leveraging traces in prompts and demonstrate that LLM-optimized prompts help outperform trace-free prompts more consistently. Additionally, we show trace-based prompting to be superior to finetuning a smaller LLM on a small-scale dataset; and conduct probing studies reinforcing the notion that execution traces can complement the reasoning abilities of the LLMs.