# CoreGen: Contextualized Code Representation Learning for Commit Message Generation

arXiv: https://arxiv.org/abs/2007.06934 · HTML: https://mlsanity.com/paper/2007.06934/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jul 2020.

## Built on by (most cited first, 15 of 15)

- [Automatic Programming: Large Language Models and Beyond](https://mlsanity.com/paper/2405.02213/) (2024)
- [Code Structure Guided Transformer for Source Code Summarization](https://mlsanity.com/paper/2104.09340/) (2021)
- [RACE: Retrieval-Augmented Commit Message Generation](https://mlsanity.com/paper/2203.02700/) (2022)
- [A Survey on Machine Learning Techniques for Source Code Analysis](https://mlsanity.com/paper/2110.09610/) (2021)
- [CommitBERT: Commit Message Generation Using Pre-Trained Programming Language Model](https://mlsanity.com/paper/2105.14242/) (2021)
- [From Commit Message Generation to History-Aware Commit Message Completion](https://mlsanity.com/paper/2308.07655/) (2023)
- [KADEL: Knowledge-Aware Denoising Learning for Commit Message Generation](https://mlsanity.com/paper/2401.08376/) (2024)
- [Towards an Understanding of Context Utilization in Code Intelligence](https://mlsanity.com/paper/2504.08734/) (2025)
- [CommitBench: A Benchmark for Commit Message Generation](https://mlsanity.com/paper/2403.05188/) (2024)
- [COMET: Generating Commit Messages using Delta Graph Context Representation](https://mlsanity.com/paper/2402.01841/) (2024)
- [Jointly Learning to Repair Code and Generate Commit Message](https://mlsanity.com/paper/2109.12296/) (2021)
- [A Full-fledged Commit Message Quality Checker Based on Machine Learning](https://mlsanity.com/paper/2309.04797/) (2023)
- [Just-In-Time Software Defect Prediction via Bi-modal Change Representation Learning](https://mlsanity.com/paper/2410.12107/) (2024)
- [A Query-Aware Multi-Path Knowledge Graph Fusion Approach for Enhancing Retrieval-Augmented Generation in Large Language Models](https://mlsanity.com/paper/2507.16826/) (2025)
- [CoRaCMG: Contextual Retrieval-Augmented Framework for Commit Message Generation](https://mlsanity.com/paper/2509.18337/) (2025)

## Abstract (arXiv)

Automatic generation of high-quality commit messages for code commits can substantially facilitate software developers' works and coordination. However, the semantic gap between source code and natural language poses a major challenge for the task. Several studies have been proposed to alleviate the challenge but none explicitly involves code contextual information during commit message generation. Specifically, existing research adopts static embedding for code tokens, which maps a token to the same vector regardless of its context. In this paper, we propose a novel Contextualized code representation learning strategy for commit message Generation (CoreGen). CoreGen first learns contextualized code representations which exploit the contextual information behind code commit sequences. The learned representations of code commits built upon Transformer are then fine-tuned for downstream commit message generation. Experiments on the benchmark dataset demonstrate the superior effectiveness of our model over the baseline models with at least 28.18% improvement in terms of BLEU-4 score. Furthermore, we also highlight the future opportunities in training contextualized code representations on larger code corpus as a solution to low-resource tasks and adapting the contextualized code representation framework to other code-to-text generation tasks.