# Delving into Parameter-Efficient Fine-Tuning in Code Change Learning: An Empirical Study

arXiv: https://arxiv.org/abs/2402.06247 · HTML: https://mlsanity.com/paper/2402.06247/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Feb 2024.

## Built on by (most cited first, 8 of 8)

- [A Survey on LoRA of Large Language Models](https://mlsanity.com/paper/2407.11046/) (2024)
- [Router-Tuning: A Simple and Effective Approach for Enabling Dynamic-Depth in Transformers](https://mlsanity.com/paper/2410.13184/) (2024)
- [OccamLLM: Fast and Exact Language Model Arithmetic in a Single Step](https://mlsanity.com/paper/2406.06576/) (2024)
- [Exploring Data-Efficient Adaptation of Large Language Models for Code Generation](https://mlsanity.com/paper/2403.00046/) (2024)
- [MSPLoRA: A Multi-Scale Pyramid Low-Rank Adaptation for Efficient Model Fine-Tuning](https://mlsanity.com/paper/2503.21838/) (2025)
- [Just-In-Time Software Defect Prediction via Bi-modal Change Representation Learning](https://mlsanity.com/paper/2410.12107/) (2024)
- [MergeRepair: An Exploratory Study on Merging Task-Specific Adapters in Code LLMs for Automated Program Repair](https://mlsanity.com/paper/2408.09568/) (2024)
- [CodeFlowLM: Incremental Just-In-Time Defect Prediction with Pretrained Language Models and Exploratory Insights into Defect Localization](https://mlsanity.com/paper/2512.00231/) (2025)

## Abstract (arXiv)

Compared to Full-Model Fine-Tuning (FMFT), Parameter Efficient Fine-Tuning (PEFT) has demonstrated superior performance and lower computational overhead in several code understanding tasks, such as code summarization and code search. This advantage can be attributed to PEFT's ability to alleviate the catastrophic forgetting issue of Pre-trained Language Models (PLMs) by updating only a small number of parameters. As a result, PEFT effectively harnesses the pre-trained general-purpose knowledge for downstream tasks. However, existing studies primarily involve static code comprehension, aligning with the pre-training paradigm of recent PLMs and facilitating knowledge transfer, but they do not account for dynamic code changes. Thus, it remains unclear whether PEFT outperforms FMFT in task-specific adaptation for code-change-related tasks. To address this question, we examine two prevalent PEFT methods, namely Adapter Tuning (AT) and Low-Rank Adaptation (LoRA), and compare their performance with FMFT on five popular PLMs. Specifically, we evaluate their performance on two widely-studied code-change-related tasks: Just-In-Time Defect Prediction (JIT-DP) and Commit Message Generation (CMG). The results demonstrate that both AT and LoRA achieve state-of-the-art (SOTA) results in JIT-DP and exhibit comparable performances in CMG when compared to FMFT and other SOTA approaches. Furthermore, AT and LoRA exhibit superiority in cross-lingual and low-resource scenarios. We also conduct three probing tasks to explain the efficacy of PEFT techniques on JIT-DP and CMG tasks from both static and dynamic perspectives. The study indicates that PEFT, particularly through the use of AT and LoRA, offers promising advantages in code-change-related tasks, surpassing FMFT in certain aspects.