# Debugging with Open-Source Large Language Models: An Evaluation

arXiv: https://arxiv.org/abs/2409.03031 · HTML: https://mlsanity.com/paper/2409.03031/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Sep 2024.

## Built on by (most cited first, 7 of 7)

- [Novice Developers' Perspectives on Adopting LLMs for Software Development: A Systematic Literature Review](https://mlsanity.com/paper/2503.07556/) (2025)
- [Synergizing LLMs and Knowledge Graphs: A Novel Approach to Software Repository-Related Question Answering](https://mlsanity.com/paper/2412.03815/) (2024)
- [Impact of Comments on LLM Comprehension of Legacy Code](https://mlsanity.com/paper/2506.11007/) (2025)
- [RAT: RunAnyThing via Fully Automated Environment Configuration](https://mlsanity.com/paper/2604.23190/) (2026)
- [LLM-Based Automated Diagnosis Of Integration Test Failures At Google](https://mlsanity.com/paper/2604.12108/) (2026)
- [uGen: An Agentic Framework for Generating Microarchitectural Attack PoCs](https://mlsanity.com/paper/2605.15503/) (2026)
- [Reasoning Beyond Prediction: From Data-Driven to Causal Software Engineering](https://mlsanity.com/paper/2606.27960/) (2026)

## Abstract (arXiv)

Large language models have shown good potential in supporting software development tasks. This is why more and more developers turn to LLMs (e.g. ChatGPT) to support them in fixing their buggy code. While this can save time and effort, many companies prohibit it due to strict code sharing policies. To address this, companies can run open-source LLMs locally. But until now there is not much research evaluating the performance of open-source large language models in debugging. This work is a preliminary evaluation of the capabilities of open-source LLMs in fixing buggy code. The evaluation covers five open-source large language models and uses the benchmark DebugBench which includes more than 4000 buggy code instances written in Python, Java and C++. Open-source LLMs achieved scores ranging from 43.9% to 66.6% with DeepSeek-Coder achieving the best score for all three programming languages.