# CommitBERT: Commit Message Generation Using Pre-Trained Programming Language Model

arXiv: https://arxiv.org/abs/2105.14242 · HTML: https://mlsanity.com/paper/2105.14242/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 28 references that are in the map.
- First posted on arXiv 29 May 2021.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [Automatically Generating Commit Messages from Diffs using Neural Machine Translation](https://mlsanity.com/paper/1708.09492/) (2017)
- [ATOM: Commit Message Generation Based on Abstract Syntax Tree and Hybrid Ranking](https://mlsanity.com/paper/1912.02972/) (2019)
- [A Neural Architecture for Generating Natural Language Descriptions from Source Code Changes](https://mlsanity.com/paper/1704.04856/) (2017)
- [CoreGen: Contextualized Code Representation Learning for Commit Message Generation](https://mlsanity.com/paper/2007.06934/) (2020)
- [Towards Automatic Generation of Short Summaries of Commits](https://mlsanity.com/paper/1703.09603/) (2017)
- [Generating Commit Messages from Git Diffs](https://mlsanity.com/paper/1911.11690/) (2019)

## Built on by (most cited first, 16 of 16)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [OctoPack: Instruction Tuning Code Large Language Models](https://mlsanity.com/paper/2308.07124/) (2023)
- [Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code](https://mlsanity.com/paper/2311.07989/) (2023)
- [Automatic Programming: Large Language Models and Beyond](https://mlsanity.com/paper/2405.02213/) (2024)
- [RACE: Retrieval-Augmented Commit Message Generation](https://mlsanity.com/paper/2203.02700/) (2022)
- [From Commit Message Generation to History-Aware Commit Message Completion](https://mlsanity.com/paper/2308.07655/) (2023)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [CommitBench: A Benchmark for Commit Message Generation](https://mlsanity.com/paper/2403.05188/) (2024)
- [CommitBART: A Large Pre-trained Model for GitHub Commits](https://mlsanity.com/paper/2208.08100/) (2022)
- [Hierarchical Repository-Level Code Summarization for Business Applications Using Local LLMs](https://mlsanity.com/paper/2501.07857/) (2025)
- [COMET: Generating Commit Messages using Delta Graph Context Representation](https://mlsanity.com/paper/2402.01841/) (2024)
- [Automatic Pull Request Description Generation Using LLMs: A T5 Model Approach](https://mlsanity.com/paper/2408.00921/) (2024)
- [Edisum: Summarizing and Explaining Wikipedia Edits at Scale](https://mlsanity.com/paper/2404.03428/) (2024)
- [Can We Generate Shellcodes via Natural Language? An Empirical Study](https://mlsanity.com/paper/2202.03755/) (2022)
- [XSearch: Explainable Code Search via Concept-to-Code Alignment](https://mlsanity.com/paper/2605.16046/) (2026)
- [Stylometric Defenses Against Author Impersonation in Software Repositories](https://mlsanity.com/paper/2608.02695/) (2026)

## Abstract (arXiv)

Commit message is a document that summarizes source code changes in natural language. A good commit message clearly shows the source code changes, so this enhances collaboration between developers. Therefore, our work is to develop a model that automatically writes the commit message. To this end, we release 345K datasets consisting of code modification and commit messages in six programming languages (Python, PHP, Go, Java, JavaScript, and Ruby). Similar to the neural machine translation (NMT) model, using our dataset, we feed the code modification to the encoder input and the commit message to the decoder input and measure the result of the generated commit message with BLEU-4. Also, we propose the following two training methods to improve the result of generating the commit message: (1) A method of preprocessing the input to feed the code modification to the encoder input. (2) A method that uses an initial weight suitable for the code domain to reduce the gap in contextual representation between programming language (PL) and natural language (NL). Training code, dataset, and pre-trained weights are available at https://github.com/graykode/commit-autosuggestions