# Using Large Language Models for Commit Message Generation: A Preliminary Study

arXiv: https://arxiv.org/abs/2401.05926 · HTML: https://mlsanity.com/paper/2401.05926/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jan 2024.

## Built on by (most cited first, 10 of 10)

- [AlignCoder: Aligning Retrieval with Target Intent for Repository-Level Code Completion](https://mlsanity.com/paper/2601.19697/) (2026)
- [Using LLMs in Generating Design Rationale for Software Architecture Decisions](https://mlsanity.com/paper/2504.20781/) (2025)
- [DRAINCODE: Stealthy Energy Consumption Attacks on Retrieval-Augmented Code Generation via Context Poisoning](https://mlsanity.com/paper/2601.20615/) (2026)
- [Evaluating LLM-Based Test Generation Under Software Evolution](https://mlsanity.com/paper/2603.23443/) (2026)
- [PhoenixRepair: Rethinking Repair Strategy Exploration in Software Agents](https://mlsanity.com/paper/2607.18859/) (2026)
- [CodeFuse-CommitEval: Towards Benchmarking LLM's Power on Commit Message and Code Change Inconsistency Detection](https://mlsanity.com/paper/2511.19875/) (2025)
- [Towards Realistic Evaluation of Commit Message Generation by Matching Online and Offline Settings](https://mlsanity.com/paper/2410.12046/) (2024)
- [AI-Powered Commit Explorer (APCE)](https://mlsanity.com/paper/2507.16063/) (2025)
- [CoRaCMG: Contextual Retrieval-Augmented Framework for Commit Message Generation](https://mlsanity.com/paper/2509.18337/) (2025)
- [Prompt-Driven Code Summarization: A Systematic Literature Review](https://mlsanity.com/paper/2604.15385/) (2026)

## Abstract (arXiv)

A commit message is a textual description of the code changes in a commit, which is a key part of the Git version control system (VCS). It captures the essence of software updating. Therefore, it can help developers understand code evolution and facilitate efficient collaboration between developers. However, it is time-consuming and labor-intensive to write good and valuable commit messages. Some researchers have conducted extensive studies on the automatic generation of commit messages and proposed several methods for this purpose, such as generationbased and retrieval-based models. However, seldom studies explored whether large language models (LLMs) can be used to generate commit messages automatically and effectively. To this end, this paper designed and conducted a series of experiments to comprehensively evaluate the performance of popular open-source and closed-source LLMs, i.e., Llama 2 and ChatGPT, in commit message generation. The results indicate that considering the BLEU and Rouge-L metrics, LLMs surpass the existing methods in certain indicators but lag behind in others. After human evaluations, however, LLMs show a distinct advantage over all these existing methods. Especially, in 78% of the 366 samples, the commit messages generated by LLMs were evaluated by humans as the best. This work not only reveals the promising potential of using LLMs to generate commit messages, but also explores the limitations of commonly used metrics in evaluating the quality of auto-generated commit messages.