# Can Large Language Models Serve as Evaluators for Code Summarization?

arXiv: https://arxiv.org/abs/2412.01333 · HTML: https://mlsanity.com/paper/2412.01333/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Dec 2024.

## Built on by (most cited first, 12 of 12)

- [Evaluating Text Creativity across Diverse Domains: A Dataset and Large Language Model Evaluator](https://mlsanity.com/paper/2505.19236/) (2025)
- [Bridging Code Graphs and Large Language Models for Better Code Understanding](https://mlsanity.com/paper/2512.07666/) (2025)
- [Agent4cs: A Multi-agent System for Code Summarization in Large Hierarchical Codebases](https://mlsanity.com/paper/2607.01425/) (2026)
- [Can Large Language Models Understand Intermediate Representations in Compilers?](https://mlsanity.com/paper/2502.06854/) (2025)
- [Knowledge-Graph-Driven Data Synthesis for Low-Resource Software Development: A HarmonyOS Case Study](https://mlsanity.com/paper/2512.00380/) (2025)
- [Issue Retrieval and Verification Enhanced Supplementary Code Comment Generation](https://mlsanity.com/paper/2506.14649/) (2025)
- [evalSmarT: An LLM-Based Framework for Evaluating Smart Contract Generated Comments](https://mlsanity.com/paper/2507.20774/) (2025)
- [Assertion-Aware Test Code Summarization with Large Language Models](https://mlsanity.com/paper/2511.06227/) (2025)
- [Precision in Practice: Knowledge Guided Code Summarizing Grounded in Industrial Expectations](https://mlsanity.com/paper/2602.03400/) (2026)
- [More than a Judge: An Empirical Study of Agent-Human Interaction in Crowdsourced Testing Assessment](https://mlsanity.com/paper/2606.06301/) (2026)
- [CoSTAR: Data Synthesis-Driven Constraint-Aware COBOL Section Summarization for Legacy System Modernization](https://mlsanity.com/paper/2609.11332/) (2026)
- [Retrieve, Reproduce, Reveal: Dissecting Retrieval-Augmented Software Vulnerability Detection](https://mlsanity.com/paper/2609.37669/) (2026)

## Abstract (arXiv)

Code summarization facilitates program comprehension and software maintenance by converting code snippets into natural-language descriptions. Over the years, numerous methods have been developed for this task, but a key challenge remains: effectively evaluating the quality of generated summaries. While human evaluation is effective for assessing code summary quality, it is labor-intensive and difficult to scale. Commonly used automatic metrics, such as BLEU, ROUGE-L, METEOR, and BERTScore, often fail to align closely with human judgments. In this paper, we explore the potential of Large Language Models (LLMs) for evaluating code summarization. We propose CODERPE (Role-Player for Code Summarization Evaluation), a novel method that leverages role-player prompting to assess the quality of generated summaries. Specifically, we prompt an LLM agent to play diverse roles, such as code reviewer, code author, code editor, and system analyst. Each role evaluates the quality of code summaries across key dimensions, including coherence, consistency, fluency, and relevance. We further explore the robustness of LLMs as evaluators by employing various prompting strategies, including chain-of-thought reasoning, in-context learning, and tailored rating form designs. The results demonstrate that LLMs serve as effective evaluators for code summarization methods. Notably, our LLM-based evaluator, CODERPE , achieves an 81.59% Spearman correlation with human evaluations, outperforming the existing BERTScore metric by 17.27%.