# Evaluating Code Summarization Techniques: A New Metric and an Empirical Characterization

arXiv: https://arxiv.org/abs/2312.15475 · HTML: https://mlsanity.com/paper/2312.15475/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Dec 2023.

## Built on by (most cited first, 21 of 21)

- [Source Code Summarization in the Era of Large Language Models](https://mlsanity.com/paper/2407.07959/) (2024)
- [Agent READMEs: An Empirical Study of Context Files for Agentic Coding](https://mlsanity.com/paper/2511.12884/) (2025)
- [Calibration of Large Language Models on Code Summarization](https://mlsanity.com/paper/2404.19318/) (2024)
- [Enhancing LLM Performance Through Debate: An Empirical Study on Multi-Agent Debate for Coding Tasks](https://mlsanity.com/paper/2503.12029/) (2025)
- [Code Summarization Beyond Function Level](https://mlsanity.com/paper/2502.16704/) (2025)
- [Leveraging LLMs for Legacy Code Modernization: Challenges and Opportunities for LLM-Generated Documentation](https://mlsanity.com/paper/2411.14971/) (2024)
- [MALSIGHT: Exploring Malicious Source Code and Benign Pseudocode for Iterative Binary Malware Summarization](https://mlsanity.com/paper/2406.18379/) (2024)
- [Summarizing Speech: A Comprehensive Survey](https://mlsanity.com/paper/2504.08024/) (2025)
- [Teamwork makes the dream work: LLMs-Based Agents for GitHub README.MD Summarization](https://mlsanity.com/paper/2503.10876/) (2025)
- [On the Compression of Language Models for Code: An Empirical Study on CodeBERT](https://mlsanity.com/paper/2412.13737/) (2024)
- [Commenting Higher-level Code Unit: Full Code, Reduced Code, or Hierarchical Code Summarization](https://mlsanity.com/paper/2503.10737/) (2025)
- [Assessing Large Language Models in Generating RTL Design Specifications](https://mlsanity.com/paper/2512.00045/) (2025)
- [Agent4cs: A Multi-agent System for Code Summarization in Large Hierarchical Codebases](https://mlsanity.com/paper/2607.01425/) (2026)
- [Variational Prefix Tuning for Diverse and Accurate Code Summarization Using Pre-trained Language Models](https://mlsanity.com/paper/2505.09062/) (2025)
- [CIDRe: A Reference-Free Multi-Aspect Criterion for Code Comment Quality Measurement](https://mlsanity.com/paper/2505.19757/) (2025)
- [ReFEree: Reference-Free and Fine-Grained Method for Evaluating Factual Consistency in Real-World Code Summarization](https://mlsanity.com/paper/2604.10520/) (2026)
- [Simple and Effective Baselines for Code Summarisation Evaluation](https://mlsanity.com/paper/2505.19392/) (2025)
- [Issue Retrieval and Verification Enhanced Supplementary Code Comment Generation](https://mlsanity.com/paper/2506.14649/) (2025)
- [Enhancing Neural Code Representation with Additional Context](https://mlsanity.com/paper/2510.12082/) (2025)
- [EyeLayer: Integrating Human Attention Patterns into LLM-Based Code Summarization](https://mlsanity.com/paper/2602.22368/) (2026)
- [Prompt-Driven Code Summarization: A Systematic Literature Review](https://mlsanity.com/paper/2604.15385/) (2026)

## Abstract (arXiv)

Several code summarization techniques have been proposed in the literature to automatically document a code snippet or a function. Ideally, software developers should be involved in assessing the quality of the generated summaries. However, in most cases, researchers rely on automatic evaluation metrics such as BLEU, ROUGE, and METEOR. These metrics are all based on the same assumption: The higher the textual similarity between the generated summary and a reference summary written by developers, the higher its quality. However, there are two reasons for which this assumption falls short: (i) reference summaries, e.g., code comments collected by mining software repositories, may be of low quality or even outdated; (ii) generated summaries, while using a different wording than a reference one, could be semantically equivalent to it, thus still being suitable to document the code snippet. In this paper, we perform a thorough empirical investigation on the complementarity of different types of metrics in capturing the quality of a generated summary. Also, we propose to address the limitations of existing metrics by considering a new dimension, capturing the extent to which the generated summary aligns with the semantics of the documented code snippet, independently from the reference summary. To this end, we present a new metric based on contrastive learning to capture said aspect. We empirically show that the inclusion of this novel dimension enables a more effective representation of developers' evaluations regarding the quality of automatically generated summaries.