mlsanityData to 7 Oct 2026

A Comprehensive Evaluation of Parameter-Efficient Fine-Tuning on Software Engineering Tasks

arXiv 2312.15614 ↗First posted 25 Dec 2023cs.SE cs.AI cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Pre-trained models (PTMs) have achieved great success in various Software Engineering (SE) downstream tasks following the ``pre-train then fine-tune'' paradigm. As fully fine-tuning all parameters of PTMs can be computationally expensive, a widely used solution is parameter-efficient fine-tuning (PEFT), which freezes PTMs while introducing extra parameters. Though work has been done to test PEFT methods in the SE field, a comprehensive evaluation is still lacking. This paper aims to fill in this gap by evaluating the effectiveness of five PEFT methods on eight PTMs and four SE downstream tasks. For different tasks and PEFT methods, we seek answers to the following research questions: 1) Is it more effective to use PTMs trained specifically on source code, or is it sufficient to use PTMs trained on natural language text? 2) What is the impact of varying model sizes? 3) How does the model architecture affect the performance? Besides effectiveness, we also discuss the efficiency of PEFT methods, concerning the costs of required training time and GPU resource consumption. We hope that our findings can provide a deeper understanding of PEFT methods on various PTMs and SE downstream tasks. All the codes and data are available at urlhttps://github.com/zwtnju/PEFT.git.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 61 of its 89 references that are in the map. First posted on arXiv 25 Dec 2023.

Builds on

61 of 89 references.

References · 61

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  12. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
38 more
  1. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  2. 2020Knowledge Distillation: A Surveycited by 1,578
  3. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  4. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  5. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  6. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  7. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  8. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  9. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  10. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  11. 2017Learning multiple visual domains with residual adapterscited by 553
  12. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  13. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  14. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  15. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  16. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 355
  17. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  18. 2020IntelliCode Compose: Code Generation Using Transformercited by 225
  19. 2019BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysiscited by 176
  20. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
  21. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  22. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 140
  23. 2022Large Language Models Meet NL2Code: A Surveycited by 135
  24. 2022A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translationcited by 97
  25. 2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
  26. 2023Towards an Understanding of Large Language Models in Software Engineering Taskscited by 67
  27. 2023On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilotcited by 64
  28. 2022An Empirical Study of Deep Learning Models for Vulnerability Detectioncited by 44
  29. 2021Multilingual training for Software Engineeringcited by 43
  30. 2022No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligencecited by 41
  31. 2020Fast and Memory-Efficient Neural Code Completioncited by 38
  32. 2020Improved Automatic Summarization of Subroutines via Attention to File Contextcited by 34
  33. 2022On the Transferability of Pre-trained Language Models for Low-Resource Programming Languagescited by 30
  34. 2021Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translationcited by 29
  35. 2021Bridging Pre-trained Models and Downstream Tasks for Source Code Understandingcited by 27
  36. 2023An Empirical Comparison of Pre-Trained Models of Source Codecited by 25
  37. 2022Compressing Pre-trained Models of Code into 3 MBcited by 21
  38. 2023Towards Efficient Fine-tuning of Pre-trained Code Models: An Experimental Study and Beyondcited by 20

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 1.

5 papers.

  1. 2024Deeper Insights Without Updates: The Power of In-Context Learning Over Fine-Tuningcited by 20
  2. 2024SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and Composition of Expertscited by 18
  3. 2025Attention, Please! Revisiting Attentive Probing Through the Lens of Efficiencycited by 11
  4. 2025HuggingGraph: Understanding the Supply Chain of LLM Ecosystemcited by 8
  5. 2026One Model, Many Skills: Parameter-Efficient Fine-Tuning for Multitask Code Analysiscited by 0