# Bug In the Code Stack: Can LLMs Find Bugs in Large Python Code Stacks

arXiv: https://arxiv.org/abs/2406.15325 · HTML: https://mlsanity.com/paper/2406.15325/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 22 references that are in the map.
- First posted on arXiv 21 Jun 2024.

## Builds on (most cited first, 9 of 9)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Using an LLM to Help With Code Understanding](https://mlsanity.com/paper/2307.08177/) (2023)
- [LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K](https://mlsanity.com/paper/2402.05136/) (2024)
- [In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Miss](https://mlsanity.com/paper/2402.10790/) (2024)
- [CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models](https://mlsanity.com/paper/2302.00288/) (2023)
- [LLM vs. Lawyers: Identifying a Subset of Summary Judgments in a Large UK Case Law Dataset](https://mlsanity.com/paper/2403.04791/) (2024)

## Built on by (most cited first, 6 of 6)

- [On Leakage of Code Generation Evaluation Datasets](https://mlsanity.com/paper/2407.07565/) (2024)
- [Assessing the Impact of Code Changes on the Fault Localizability of Large Language Models](https://mlsanity.com/paper/2504.04372/) (2025)
- [Clotho: Measuring Task-Specific Pre-Generation Test Adequacy for LLM Inputs](https://mlsanity.com/paper/2509.17314/) (2025)
- [Enhancing Large Language Models with Retrieval Augmented Generation for Software Testing and Inspection Automation](https://mlsanity.com/paper/2604.15270/) (2026)
- [An Empirical Evaluation of Locally Deployed LLMs for Bug Detection in Python Code](https://mlsanity.com/paper/2604.23361/) (2026)
- [Code Understanding is a Bottleneck for Coding Agents](https://mlsanity.com/paper/2610.10610/) (2026)

## Abstract (arXiv)

Recent research in Needle-in-a-Haystack (NIAH) benchmarks has explored the capabilities of Large Language Models (LLMs) in retrieving contextual information from large text documents. However, as LLMs become increasingly integrated into software development processes, it is crucial to evaluate their performance in code-based environments. As LLMs are further developed for program synthesis, we need to ensure that LLMs can understand syntax and write syntactically correct code. As a step in ensuring LLMs understand syntax, LLMs can be evaluated in their ability to find and detect syntax bugs. Our benchmark, Bug In The Code Stack (BICS), is designed to assess the ability of LLMs to identify simple syntax bugs within large source code. Our findings reveal three key insights: (1) code-based environments pose significantly more challenge compared to text-based environments for retrieval tasks, (2) there is a substantial performance disparity among different models, and (3) there is a notable correlation between longer context lengths and performance degradation, though the extent of this degradation varies between models.