# FeedbackEval: A Benchmark for Evaluating Large Language Models in Feedback-Driven Code Repair Tasks

arXiv: https://arxiv.org/abs/2504.06939 · HTML: https://mlsanity.com/paper/2504.06939/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Apr 2025.

## Built on by (most cited first, 10 of 10)

- [Code as Agent Harness](https://mlsanity.com/paper/2605.18747/) (2026)
- [The Rise of AI Teammates in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshaping Software Engineering](https://mlsanity.com/paper/2507.15003/) (2025)
- [RepoDebug: Repository-Level Multi-Task and Multi-Language Debugging Evaluation of Large Language Models](https://mlsanity.com/paper/2509.04078/) (2025)
- [Agentic MIP Research: Accelerated Constraint Handler Generation](https://mlsanity.com/paper/2605.09186/) (2026)
- [Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair](https://mlsanity.com/paper/2607.24604/) (2026)
- [Invalidation Contracts for Cross-Episode Agent Memory](https://mlsanity.com/paper/2609.00243/) (2026)
- [PYTHALAB-MERA: Validation-Grounded Memory, Retrieval, and Acceptance Control for Frozen-LLM Coding Agents](https://mlsanity.com/paper/2605.08468/) (2026)
- [WatchPoint: Executable User Feedback for Real-World Agentic Web Development](https://mlsanity.com/paper/2609.26204/) (2026)
- [Security in LLM-Assisted Coding: A Systematic Literature Review](https://mlsanity.com/paper/2610.15482/) (2026)
- [LintEval: Benchmarking LLMs on Linter-Guided File Refactoring](https://mlsanity.com/paper/2610.20027/) (2026)

## Abstract (arXiv)

Code repair is a fundamental task in software development, facilitating efficient bug resolution and software maintenance. Although large language models (LLMs) have demonstrated considerable potential in automated code repair, their ability to comprehend and leverage diverse types of feedback, which is crucial for iterative self-correction in authentic debugging scenarios, remains insufficiently understood. To bridge this gap, we introduce FeedbackEval, a systematic benchmark constructed from three heterogeneous sources (HumanEval, CoderEval, and SWE-Bench-verified), to evaluate LLMs' feedback comprehension and code repair performance. We conduct a comprehensive empirical study on five state-of-the-art LLMs, including GPT-4o, Claude-3.5, Deepseek-R1, GLM-4, and Qwen2.5, to evaluate their behavior under both single-iteration and iterative code repair settings. Our results show that mixed feedback yields the highest repair success (63.6%), with LLM-Expert and test feedback providing strong targeted gains (62.9% and 57.9%, respectively), while minimal (53.1%) and compiler feedback (49.2%) offer moderate benefits and LLM-Skilled proves least effective (48.8%). Iterative feedback further enhances repair performance, though the marginal benefit diminishes after two or three iterations. Moreover, prompt structure is shown to be critical: structured reasoning (RR, CoT) and dynamic example selection deliver notable improvements, whereas removing semantic cues such as docstrings or role-play causes severe degradation. This work introduces a robust benchmark and delivers practical insights to advance the understanding and development of feedback-driven code repair using LLMs.