# Exploring the Capabilities of LLMs for Code Change Related Tasks

arXiv: https://arxiv.org/abs/2407.02824 · HTML: https://mlsanity.com/paper/2407.02824/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jul 2024.

## Built on by (most cited first, 18 of 18)

- [On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub](https://mlsanity.com/paper/2509.14745/) (2025)
- [Bridging the Editing Gap in LLMs: FineEdit for Precise and Targeted Text Modifications](https://mlsanity.com/paper/2502.13358/) (2025)
- [Top General Performance = Top Domain Performance? DomainCodeBench: A Multi-domain Code Generation Benchmark](https://mlsanity.com/paper/2412.18573/) (2024)
- [CRScore: Grounding Automated Evaluation of Code Review Comments in Code Claims and Smells](https://mlsanity.com/paper/2409.19801/) (2024)
- [HyperEdit: Unlocking Instruction-based Text Editing in LLMs via Hypernetworks](https://mlsanity.com/paper/2512.12544/) (2025)
- [Repo0: Design-Driven Zero-to-All Code Generation](https://mlsanity.com/paper/2608.19854/) (2026)
- [CODE-DITING: A Reasoning-Based Metric for Functional Alignment in Code Evaluation](https://mlsanity.com/paper/2505.19502/) (2025)
- [Augmenting Large Language Models with Static Code Analysis for Automated Code Quality Improvements](https://mlsanity.com/paper/2506.10330/) (2025)
- [LLM-Driven Collaborative Model for Untangling Commits via Explicit and Implicit Dependency Reasoning](https://mlsanity.com/paper/2507.16395/) (2025)
- [JSON Whisperer: Efficient JSON Editing with LLMs](https://mlsanity.com/paper/2510.04717/) (2025)
- [To Diff or Not to Diff? Structure-Aware and Adaptive Output Formats for Efficient LLM-based Code Editing](https://mlsanity.com/paper/2604.27296/) (2026)
- [Role-RL: Online Long-Context Processing with Role Reinforcement Learning for Distinct LLMs in Their Optimal Roles](https://mlsanity.com/paper/2409.18014/) (2024)
- [What a diff makes: automating code migration with large language models](https://mlsanity.com/paper/2511.00160/) (2025)
- [A Dataset and Preliminary Study of Using GPT-5 for Code-change Impact Analysis](https://mlsanity.com/paper/2512.19481/) (2025)
- [MuMuTestUp: Mutation-based Multi-Agent Test Case Update](https://mlsanity.com/paper/2605.19265/) (2026)
- [Diff-Based Code Corruption using LLMs for Large-Scale Bugfix Benchmarking](https://mlsanity.com/paper/2606.29088/) (2026)
- [SynH-Rank: Quality-Aware Code Search via Diverse Data Synthesis and Hierarchical Ranking Training](https://mlsanity.com/paper/2607.17139/) (2026)
- [Using Agentic AI for contextualized and multifaceted code review at Ericsson](https://mlsanity.com/paper/2609.15877/) (2026)

## Abstract (arXiv)

Developers deal with code-change-related tasks daily, e.g., reviewing code. Pre-trained code and code-change-oriented models have been adapted to help developers with such tasks. Recently, large language models (LLMs) have shown their effectiveness in code-related tasks. However, existing LLMs for code focus on general code syntax and semantics rather than the differences between two code versions. Thus, it is an open question how LLMs perform on code-change-related tasks. To answer this question, we conduct an empirical study using textgreater 1B parameters LLMs on three code-change-related tasks, i.e., code review generation, commit message generation, and just-in-time comment update, with in-context learning (ICL) and parameter-efficient fine-tuning (PEFT, including LoRA and prefix-tuning). We observe that the performance of LLMs is poor without examples and generally improves with examples, but more examples do not always lead to better performance. LLMs tuned with LoRA have comparable performance to the state-of-the-art small pre-trained models. Larger models are not always better, but textscLlama~2 and textscCode~Llama families are always the best. The best LLMs outperform small pre-trained models on the code changes that only modify comments and perform comparably on other code changes. We suggest future work should focus more on guiding LLMs to learn the knowledge specific to the changes related to code rather than comments for code-change-related tasks.