# Identifying Inaccurate Descriptions in LLM-generated Code Comments via Test Execution

arXiv: https://arxiv.org/abs/2406.14836 · HTML: https://mlsanity.com/paper/2406.14836/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jun 2024.

## Built on by (most cited first, 8 of 8)

- [ETF: An Entity Tracing Framework for Hallucination Detection in Code Summaries](https://mlsanity.com/paper/2410.14748/) (2024)
- [Hallucinations in Code Change to Natural Language Generation: Prevalence and Evaluation of Detection Metrics](https://mlsanity.com/paper/2508.08661/) (2025)
- [ReFEree: Reference-Free and Fine-Grained Method for Evaluating Factual Consistency in Real-World Code Summarization](https://mlsanity.com/paper/2604.10520/) (2026)
- [Characterizing Readability Issue Patterns and the Role of Prompt Design in LLM-Generated Code](https://mlsanity.com/paper/2605.13280/) (2026)
- [Automated Harmfulness Testing for Code Large Language Models](https://mlsanity.com/paper/2503.16740/) (2025)
- [XSearch: Explainable Code Search via Concept-to-Code Alignment](https://mlsanity.com/paper/2605.16046/) (2026)
- [The Hitchhiker's Guide to Monoculture: AI Homogenizes Syntax, Not (Necessarily) Semantics](https://mlsanity.com/paper/2607.13077/) (2026)
- [Talking to Itself While Coding: What Makes Comments Help Code Generation?](https://mlsanity.com/paper/2609.09242/) (2026)

## Abstract (arXiv)

Software comments are critical for human understanding of software, and as such many comment generation techniques have been proposed. However, we find that a systematic evaluation of the factual accuracy of generated comments is rare; only subjective accuracy labels have been given. Evaluating comments generated by three Large Language Models (LLMs), we find that even for the best-performing LLM, roughly a fifth of its comments contained demonstrably inaccurate statements. While it seems code-comment consistency detection techniques should be able to detect inaccurate comments, we perform experiments demonstrating they have no statistically significant relationship with comment accuracy, underscoring the substantial difficulty of this problem. To tackle this, we propose the concept of document testing, in which a document is verified by using an LLM to generate tests based on the document, running those tests, and observing whether they pass or fail. Furthermore, we implement our concept to verify Java comments. Experiments demonstrate that our approach has a robust statistical relationship with comment accuracy, making headway into a problem where prior techniques failed. Qualitative evaluation also reveals the promise of our approach in gaining developer trust, while highlighting the limitations of our current implementation.