# Impact of Comments on LLM Comprehension of Legacy Code

arXiv: https://arxiv.org/abs/2506.11007 · HTML: https://mlsanity.com/paper/2506.11007/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 15 references that are in the map.
- First posted on arXiv 23 Apr 2025.

## Builds on (most cited first, 12 of 12)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Using an LLM to Help With Code Understanding](https://mlsanity.com/paper/2307.08177/) (2023)
- [Large Language Models for Software Engineering: Survey and Open Problems](https://mlsanity.com/paper/2310.03533/) (2023)
- [DebugBench: Evaluating Debugging Capability of Large Language Models](https://mlsanity.com/paper/2401.04621/) (2024)
- [Can multiple-choice questions really be useful in detecting the abilities of LLMs?](https://mlsanity.com/paper/2403.17752/) (2024)
- [Comparing Code Explanations Created by Students and Large Language Models](https://mlsanity.com/paper/2304.03938/) (2023)
- [Automatic Code Documentation Generation Using GPT-3](https://mlsanity.com/paper/2209.02235/) (2022)
- [LLMs May Perform MCQA by Selecting the Least Incorrect Option](https://mlsanity.com/paper/2402.01349/) (2024)
- [Multiple-Choice Questions are Efficient and Robust LLM Evaluators](https://mlsanity.com/paper/2405.11966/) (2024)
- [A Comparative Analysis of Large Language Models for Code Documentation Generation](https://mlsanity.com/paper/2312.10349/) (2023)
- [Debugging with Open-Source Large Language Models: An Evaluation](https://mlsanity.com/paper/2409.03031/) (2024)
- [Leveraging LLMs for Legacy Code Modernization: Challenges and Opportunities for LLM-Generated Documentation](https://mlsanity.com/paper/2411.14971/) (2024)

## Built on by (most cited first, 1 of 1)

- [Spec2COBOLRot: An Agentic-AI Degradation Loop for Realistic COBOL Corpus Generation](https://mlsanity.com/paper/2609.26835/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have been increasingly integrated into software engineering and maintenance tasks due to their high performance with software engineering tasks and robust understanding of modern programming languages. However, the ability of LLMs to comprehend code written with legacy languages remains a research gap challenged by real-world legacy systems lacking or containing inaccurate documentation that may impact LLM comprehension. To assess LLM comprehension of legacy languages, there is a need for objective LLM evaluation. In order to objectively measure LLM comprehension of legacy languages, we need an efficient, quantitative evaluation method. We leverage multiple-choice question answering (MCQA), an emerging LLM evaluation methodology, to evaluate LLM comprehension of legacy code and the impact of comment prevalence and inaccurate comments. In this work, we present preliminary findings on the impact of documentation on LLM comprehension of legacy code and outline strategic objectives for future work.