# Improving LLM Classification of Logical Errors by Integrating Error Relationship into Prompts

arXiv: https://arxiv.org/abs/2404.19336 · HTML: https://mlsanity.com/paper/2404.19336/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 15 references that are in the map.
- First posted on arXiv 30 Apr 2024.

## Builds on (most cited first, 9 of 9)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Large Language Models: A Survey](https://mlsanity.com/paper/2402.06196/) (2024)
- [Automatic Generation of Programming Exercises and Code Explanations using Large Language Models](https://mlsanity.com/paper/2206.11861/) (2022)
- [Graph-based, Self-Supervised Program Repair from Diagnostic Feedback](https://mlsanity.com/paper/2005.10636/) (2020)
- [Less Training, More Repairing Please: Revisiting Automated Program Repair via Zero-shot Learning](https://mlsanity.com/paper/2207.08281/) (2022)
- [Using Large Language Models to Enhance Programming Error Messages](https://mlsanity.com/paper/2210.11630/) (2022)
- [Decoding Logic Errors: A Comparative Study on Bug Detection by Students and Large Language Models](https://mlsanity.com/paper/2311.16017/) (2023)

## Built on by (most cited first, 5 of 5)

- [CASTLE: Benchmarking Dataset for Static Code Analyzers and LLMs towards CWE Detection](https://mlsanity.com/paper/2503.09433/) (2025)
- [Automated Identification of Logical Errors in Programs: Advancing Scalable Analysis of Student Misconceptions](https://mlsanity.com/paper/2505.10913/) (2025)
- [Benchmarking the rationality of AI decision making using the transitivity axiom](https://mlsanity.com/paper/2502.10554/) (2025)
- [McMining: Automated Discovery of Misconceptions in Student Code](https://mlsanity.com/paper/2510.08827/) (2025)
- [Plausibility as Failure: How LLMs and Humans Co-Construct Epistemic Error](https://mlsanity.com/paper/2512.16750/) (2025)

## Abstract (arXiv)

LLMs trained in the understanding of programming syntax are now providing effective assistance to developers and are being used in programming education such as in generation of coding problem examples or providing code explanations. A key aspect of programming education is understanding and dealing with error message. However, 'logical errors' in which the program operates against the programmer's intentions do not receive error messages from the compiler. In this study, building on existing research on programming errors, we first define the types of logical errors that can occur in programming in general. Based on the definition, we propose an effective approach for detecting logical errors with LLMs that makes use of relations among error types in the Chain-of-Thought and Tree-of-Thought prompts. The experimental results indicate that when such logical error descriptions in the prompt are used, the average classifition performance is about 21% higher than the ones without them. We also conducted an experiment for exploiting the relations among errors in generating a new logical error dataset using LLMs. As there is very limited dataset for logical errors such benchmark dataset can be very useful for various programming related applications. We expect that our work can assist novice programmers in identifying the causes of code errors and correct them more effectively.