# CREF: An LLM-based Conversational Software Repair Framework for Programming Tutors

arXiv: https://arxiv.org/abs/2406.13972 · HTML: https://mlsanity.com/paper/2406.13972/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jun 2024.

## Built on by (most cited first, 15 of 15)

- [Training LLM-based Tutors to Improve Student Learning Outcomes in Dialogues](https://mlsanity.com/paper/2503.06424/) (2025)
- [EduBench: A Comprehensive Benchmarking Dataset for Evaluating Large Language Models in Diverse Educational Scenarios](https://mlsanity.com/paper/2505.16160/) (2025)
- [Repair-R1: Better Test Before Repair](https://mlsanity.com/paper/2507.22853/) (2025)
- [HLSDebugger: Identification and Correction of Logic Bugs in HLS Code with LLM Solutions](https://mlsanity.com/paper/2507.21485/) (2025)
- [Integrating Various Software Artifacts for Better LLM-based Bug Localization and Program Repair](https://mlsanity.com/paper/2412.03905/) (2024)
- [Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Quality](https://mlsanity.com/paper/2503.05860/) (2025)
- [When Models Edit Too Much: On the Fidelity of Minimal Code Edits](https://mlsanity.com/paper/2609.04061/) (2026)
- [FailureMem: A Failure-Aware Multimodal Framework for Autonomous Software Repair](https://mlsanity.com/paper/2603.17826/) (2026)
- [Collaborative Agents for Automated Program Repair in Ruby](https://mlsanity.com/paper/2511.03925/) (2025)
- [Multi-Agent Systems for Dataset Adaptation in Software Engineering: Capabilities, Limitations, and Future Directions](https://mlsanity.com/paper/2511.21380/) (2025)
- [Evaluating adaptive and generative AI-based feedback and recommendations in a knowledge-graph-integrated programming learning system](https://mlsanity.com/paper/2603.24940/) (2026)
- [DebugTA: An LLM-Based Agent for Simplifying Debugging and Teaching in Programming Education](https://mlsanity.com/paper/2510.11076/) (2025)
- [From Pilots to Practices: A Scoping Review of GenAI-Enabled Personalization in Computer Science Education](https://mlsanity.com/paper/2512.20714/) (2025)
- [A Tool-Augmented, GPT-4 Chatbot for Real-Time Repository Data Analysis](https://mlsanity.com/paper/2609.07586/) (2026)
- [Security in LLM-Assisted Coding: A Systematic Literature Review](https://mlsanity.com/paper/2610.15482/) (2026)

## Abstract (arXiv)

Program repair techniques offer cost-saving benefits for debugging within software development and programming education scenarios. With the proven effectiveness of Large Language Models (LLMs) in code-related tasks, researchers have explored their potential for program repair. However, it is crucial to recognize that existing repair benchmarks may have influenced LLM training data, potentially causing data leakage. To evaluate LLMs' realistic repair capabilities, (1) we introduce an extensive, non-crawled benchmark, referred to as TutorCode, comprising 1,239 C++ defect codes and associated information such as tutor guidance, solution description, failing test cases, and the corrected code. Our work assesses the repair performance of 12 LLMs on TutorCode, measuring repair correctness (TOP-5 and AVG-5) and patch precision (RPSR). (2) We then provide a comprehensive investigation into which types of extra information can help LLMs improve their performance in repairing defects. Among these types, tutor guidance was found to be the most effective information in enhancing LLM repair capabilities. To fully harness LLMs' conversational capabilities and the benefits of augmented information, (3) we introduce a novel conversational semi-automatic repair framework CREF assisting human tutor. It demonstrates a remarkable AVG-5 improvement of 17.2%-24.6% compared to the baseline, achieving an impressive AVG-5 of 76.6% when utilizing GPT-4. These results highlight the potential for enhancing LLMs' repair capabilities through interactions with tutors and historical conversations involving incorrect responses. The successful application of CREF in a real-world educational setting demonstrates its effectiveness in reducing tutors' workload and improving students' learning experience, while also showcasing its promise for facilitating other software engineering tasks, such as code review.