# Evaluating the Quality of Code Comments Generated by Large Language Models for Novice Programmers

arXiv: https://arxiv.org/abs/2409.14368 · HTML: https://mlsanity.com/paper/2409.14368/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 14 references that are in the map.
- First posted on arXiv 22 Sep 2024.

## Builds on (most cited first, 8 of 8)

- [A Systematic Evaluation of Large Language Models of Code](https://mlsanity.com/paper/2202.13169/) (2022)
- [Language Models of Code are Few-Shot Commonsense Learners](https://mlsanity.com/paper/2210.07128/) (2022)
- [Using Large Language Models to Enhance Programming Error Messages](https://mlsanity.com/paper/2210.11630/) (2022)
- [Comparing Code Explanations Created by Students and Large Language Models](https://mlsanity.com/paper/2304.03938/) (2023)
- [Explaining Competitive-Level Programming Solutions using LLMs](https://mlsanity.com/paper/2307.05337/) (2023)
- [Automatically Generating CS Learning Materials with Large Language Models](https://mlsanity.com/paper/2212.05113/) (2022)
- [CodeExp: Explanatory Code Document Generation](https://mlsanity.com/paper/2211.15395/) (2022)
- [Exploring the Potential of Large Language Models in Generating Code-Tracing Questions for Introductory Programming Courses](https://mlsanity.com/paper/2310.15317/) (2023)

## Abstract (arXiv)

Large Language Models (LLMs) show promise in generating code comments for novice programmers, but their educational effectiveness remains under-evaluated. This study assesses the instructional quality of code comments produced by GPT-4, GPT-3.5-Turbo, and Llama2, compared to expert-developed comments, focusing on their suitability for novices. Analyzing a dataset of ``easy'' level Java solutions from LeetCode, we find that GPT-4 exhibits comparable quality to expert comments in aspects critical for beginners, such as clarity, beginner-friendliness, concept elucidation, and step-by-step guidance. GPT-4 outperforms Llama2 in discussing complexity (chi-square = 11.40, p = 0.001) and is perceived as significantly more supportive for beginners than GPT-3.5 and Llama2 with Mann-Whitney U-statistics = 300.5 and 322.5, p = 0.0017 and 0.0003). This study highlights the potential of LLMs for generating code comments tailored to novice programmers.