# CCT5: A Code-Change-Oriented Pre-Trained Model

arXiv: https://arxiv.org/abs/2305.10785 · HTML: https://mlsanity.com/paper/2305.10785/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 May 2023.

## Built on by (most cited first, 18 of 18)

- [MarsCode Agent: AI-native Automated Bug Fixing](https://mlsanity.com/paper/2409.00899/) (2024)
- [An Empirical Study on LLM-based Agents for Automated Bug Fixing](https://mlsanity.com/paper/2411.10213/) (2024)
- [Model Editing for LLMs4Code: How Far are We?](https://mlsanity.com/paper/2411.06638/) (2024)
- [CodeArt: Better Code Models by Attention Regularization When Symbols Are Lacking](https://mlsanity.com/paper/2402.11842/) (2024)
- [Instruct or Interact? Exploring and Eliciting LLMs' Capability in Code Snippet Adaptation Through Prompt Engineering](https://mlsanity.com/paper/2411.15501/) (2024)
- [Large Language Models are Qualified Benchmark Builders: Rebuilding Pre-Training Datasets for Advancing Code Intelligence Tasks](https://mlsanity.com/paper/2504.19444/) (2025)
- [Smoke and Mirrors: Jailbreaking LLM-based Code Generation via Implicit Malicious Prompts](https://mlsanity.com/paper/2503.17953/) (2025)
- [Investigating Autonomous Agent Contributions in the Wild: Activity Patterns and Code Change over Time](https://mlsanity.com/paper/2604.00917/) (2026)
- [Improving Retrieval-Augmented Code Comment Generation by Retrieving for Generation](https://mlsanity.com/paper/2408.03623/) (2024)
- [Leveraging Reviewer Experience in Code Review Comment Generation](https://mlsanity.com/paper/2409.10959/) (2024)
- [Towards Practical Defect-Focused Automated Code Review](https://mlsanity.com/paper/2505.17928/) (2025)
- [LAMS: LLM-Driven Automatic Mode Switching for Assistive Teleoperation](https://mlsanity.com/paper/2501.08558/) (2025)
- [CodeLL: A Lifelong Learning Dataset to Support the Co-Evolution of Data and Language Models of Code](https://mlsanity.com/paper/2312.12492/) (2023)
- [Wired for Reuse: Automating Context-Aware Code Adaptation in IDEs via LLM-Based Agent](https://mlsanity.com/paper/2507.01315/) (2025)
- [ReDef: Do Code Language Models Truly Understand Code Changes for Just-in-Time Software Defect Prediction?](https://mlsanity.com/paper/2509.09192/) (2025)
- [A Dataset and Preliminary Study of Using GPT-5 for Code-change Impact Analysis](https://mlsanity.com/paper/2512.19481/) (2025)
- [Coding in a Bubble? Evaluating LLMs in Resolving Context Adaptation Bugs During Code Adaptation](https://mlsanity.com/paper/2601.06497/) (2026)
- [Diff-Based Code Corruption using LLMs for Large-Scale Bugfix Benchmarking](https://mlsanity.com/paper/2606.29088/) (2026)

## Abstract (arXiv)

Software is constantly changing, requiring developers to perform several derived tasks in a timely manner, such as writing a description for the intention of the code change, or identifying the defect-prone code changes. Considering that the cost of dealing with these tasks can account for a large proportion (typically around 70 percent) of the total development expenditure, automating such processes will significantly lighten the burdens of developers. To achieve such a target, existing approaches mainly rely on training deep learning models from scratch or fine-tuning existing pretrained models on such tasks, both of which have weaknesses. Specifically, the former uses comparatively small-scale labelled data for training, making it difficult to learn and exploit the domain knowledge of programming language hidden in the large-amount unlabelled code in the wild; the latter is hard to fully leverage the learned knowledge of the pre-trained model, as existing pre-trained models are designed to encode a single code snippet rather than a code change (i.e., the difference between two code snippets). We propose to pre-train a model specially designed for code changes to better support developers in software maintenance. To this end, we first collect a large-scale dataset containing 1.5M+ pairwise data of code changes and commit messages. Based on these data, we curate five different tasks for pre-training, which equip the model with diverse domain knowledge about code changes. We fine-tune the pre-trained model, CCT5, on three widely-studied tasks incurred by code changes and two tasks specific to the code review process. Results show that CCT5 outperforms both conventional deep learning approaches and existing pre-trained models on these tasks.