# On the Evaluation of Commit Message Generation Models: An Experimental Study arXiv: https://arxiv.org/abs/2107.05373 ยท HTML: https://mlsanity.com/paper/2107.05373/ ## Key facts - Cited by 20 papers in the map (data through 14 Oct 2026). - Builds on 14 of its 42 references that are in the map. - First posted on arXiv 12 Jul 2021. ## Builds on (most cited first, 14 of 14) - [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014) - [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014) - [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017) - [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015) - [code2seq: Generating Sequences from Structured Representations of Code](https://mlsanity.com/paper/1808.01400/) (2018) - [Recommendations for Datasets for Source Code Summarization](https://mlsanity.com/paper/1904.02660/) (2019) - [Automatically Generating Commit Messages from Diffs using Neural Machine Translation](https://mlsanity.com/paper/1708.09492/) (2017) - [Deep API Learning](https://mlsanity.com/paper/1605.08535/) (2016) - [CC2Vec: Distributed Representations of Code Changes](https://mlsanity.com/paper/2003.05620/) (2020) - [ATOM: Commit Message Generation Based on Abstract Syntax Tree and Hybrid Ranking](https://mlsanity.com/paper/1912.02972/) (2019) - [A Neural Architecture for Generating Natural Language Descriptions from Source Code Changes](https://mlsanity.com/paper/1704.04856/) (2017) - [Nematus: a Toolkit for Neural Machine Translation](https://mlsanity.com/paper/1703.04357/) (2017) - [Automatic Generation of Pull Request Descriptions](https://mlsanity.com/paper/1909.06987/) (2019) - [Towards Automatic Generation of Short Summaries of Commits](https://mlsanity.com/paper/1703.09603/) (2017) ## Built on by (most cited first, 20 of 20) - [On the Evaluation of Neural Code Summarization](https://mlsanity.com/paper/2107.07112/) (2021) - [RACE: Retrieval-Augmented Commit Message Generation](https://mlsanity.com/paper/2203.02700/) (2022) - [CoCoSoDa: Effective Contrastive Learning for Code Search](https://mlsanity.com/paper/2204.03293/) (2022) - [From Commit Message Generation to History-Aware Commit Message Completion](https://mlsanity.com/paper/2308.07655/) (2023) - [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025) - [KADEL: Knowledge-Aware Denoising Learning for Commit Message Generation](https://mlsanity.com/paper/2401.08376/) (2024) - [Towards an Understanding of Context Utilization in Code Intelligence](https://mlsanity.com/paper/2504.08734/) (2025) - [CommitBench: A Benchmark for Commit Message Generation](https://mlsanity.com/paper/2403.05188/) (2024) - [AlignCoder: Aligning Retrieval with Target Intent for Repository-Level Code Completion](https://mlsanity.com/paper/2601.19697/) (2026) - [CommitBART: A Large Pre-trained Model for GitHub Commits](https://mlsanity.com/paper/2208.08100/) (2022) - [Using Developer Discussions to Guide Fixing Bugs in Software](https://mlsanity.com/paper/2211.06335/) (2022) - [CCRep: Learning Code Change Representations via Pre-Trained Code Model and Query Back](https://mlsanity.com/paper/2302.03924/) (2023) - [COMET: Generating Commit Messages using Delta Graph Context Representation](https://mlsanity.com/paper/2402.01841/) (2024) - [ShortCoder: Knowledge-Augmented Syntax Optimization for Token-Efficient Code Generation](https://mlsanity.com/paper/2601.09703/) (2026) - [DRAINCODE: Stealthy Energy Consumption Attacks on Retrieval-Augmented Code Generation via Context Poisoning](https://mlsanity.com/paper/2601.20615/) (2026) - [PhoenixRepair: Rethinking Repair Strategy Exploration in Software Agents](https://mlsanity.com/paper/2607.18859/) (2026) - [CodeFuse-CommitEval: Towards Benchmarking LLM's Power on Commit Message and Code Change Inconsistency Detection](https://mlsanity.com/paper/2511.19875/) (2025) - [RefactorAssist: Agentic Refinement for Reliable Code Refactoring](https://mlsanity.com/paper/2608.00924/) (2026) - [Predicting Line-Level Defects by Capturing Code Contexts with Hierarchical Transformers](https://mlsanity.com/paper/2312.11889/) (2023) - [Towards Realistic Evaluation of Commit Message Generation by Matching Online and Offline Settings](https://mlsanity.com/paper/2410.12046/) (2024) ## Abstract (arXiv) Commit messages are natural language descriptions of code changes, which are important for program understanding and maintenance. However, writing commit messages manually is time-consuming and laborious, especially when the code is updated frequently. Various approaches utilizing generation or retrieval techniques have been proposed to automatically generate commit messages. To achieve a better understanding of how the existing approaches perform in solving this problem, this paper conducts a systematic and in-depth analysis of the state-of-the-art models and datasets. We find that: (1) Different variants of the BLEU metric are used in previous works, which affects the evaluation and understanding of existing methods. (2) Most existing datasets are crawled only from Java repositories while repositories in other programming languages are not sufficiently explored. (3) Dataset splitting strategies can influence the performance of existing models by a large margin. Some models show better performance when the datasets are split by commit, while other models perform better when the datasets are split by timestamp or by project. Based on our findings, we conduct a human evaluation and find the BLEU metric that best correlates with the human scores for the task. We also collect a large-scale, information-rich, and multi-language commit message dataset MCMD and evaluate existing models on this dataset. Furthermore, we conduct extensive experiments under different dataset splitting strategies and suggest the suitable models under different scenarios. Based on the experimental results and findings, we provide feasible suggestions for comprehensively evaluating commit message generation models and discuss possible future research directions. We believe this work can help practitioners and researchers better evaluate and select models for automatic commit message generation.