# Exploring the Potential of Conversational Test Suite Based Program Repair on SWE-bench

arXiv: https://arxiv.org/abs/2410.04485 · HTML: https://mlsanity.com/paper/2410.04485/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 11 references that are in the map.
- First posted on arXiv 6 Oct 2024.

## Builds on (most cited first, 5 of 5)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Practical Program Repair in the Era of Large Pre-trained Language Models](https://mlsanity.com/paper/2210.14179/) (2022)
- [Keep the Conversation Going: Fixing 162 out of 337 bugs for 0.42 each using ChatGPT](https://mlsanity.com/paper/2304.00385/) (2023)
- [SelfAPR: Self-supervised Program Repair with Test Execution Diagnostics](https://mlsanity.com/paper/2203.12755/) (2022)
- [A Critical Review of Large Language Model on Software Engineering: An Example from ChatGPT and Automated Program Repair](https://mlsanity.com/paper/2310.08879/) (2023)

## Built on by (most cited first, 1 of 1)

- [Dissecting the SWE-Bench Leaderboards: Profiling Submitters and Architectures of LLM- and Agent-Based Repair Systems](https://mlsanity.com/paper/2506.17208/) (2025)

## Abstract (arXiv)

Automatic program repair at project level may open yet to be seen opportunities in various fields of human activity. Since the SWE-Bench challenge was presented, we have seen numerous of solutions. Patch generation is a part of program repair, and test suite-based conversational patch generation has proven its effectiveness. However, the potential of conversational patch generation has not yet specifically estimated on SWE-Bench. This study reports experimental results aimed at evaluating the individual effectiveness of conversational patch generation on problems from SWE-Bench. The experiments show that a simple conversational pipeline based on LLaMA 3.1 70B can generate valid patches in 47% of cases, which is comparable to the state-of-the-art in program repair on SWE-Bench.