mlsanityData to 7 Oct 2026

MergeRepair: An Exploratory Study on Merging Task-Specific Adapters in Code LLMs for Automated Program Repair

arXiv 2408.09568 ↗First posted 18 Aug 2024cs.SE cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have shown high capabilities in several software development-related tasks such as program repair, documentation, code refactoring, debugging, and testing. However, training these models requires massive amount of data and significant computational resources. Adapters are specialized, small modules designed for parameter efficient fine-tuning of LLMs for specific tasks, domains, or applications without requiring extensive retraining of the entire model. These adapters offer a more efficient way to customize LLMs for particular needs, leveraging the pre-existing capabilities of the large model. Model (and adapter) merging have emerged as a technique to develop one model capable of multiple tasks, with minimal or no training required. Although model and adapter merging has shown promising performance in domains such as natural language processing and computer vision, its applicability to software engineering tasks remains underexplored. In this paper, we investigate the effectiveness of merged adapters within the context of software engineering, with a particular focus on the Automated Program Repair (APR) task, through our approach, MergeRepair. In particular, we merge multiple task-specific adapters using three different merging methods, including weight-averaging, ties, and dare-ties, and evaluate the performance of the merged adapter on the APR task. We introduce a continual merging approach, a novel method in which we sequentially merge the task-specific adapters where the order and weight of the merged adapters play a significant role. We further compare the performance of our approach with a baseline method consisting of equal-weight merging applied on parameters of different adapters, where all adapters are of equal importance.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 54 of its 87 references that are in the map. First posted on arXiv 18 Aug 2024.

Builds on

54 of 87 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  10. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  11. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  12. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
38 more
  1. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  2. 2023StarCoder: may the source be with you!cited by 790
  3. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  4. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  5. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  6. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  7. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  8. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  9. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  10. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  11. 2022Efficient Training of Language Models to Fill in the Middlecited by 248
  12. 2024Evolutionary Optimization of Model Merging Recipescited by 204
  13. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 189
  14. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  15. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  16. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 155
  17. 2018SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repaircited by 144
  18. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  19. 2022Practical Program Repair in the Era of Large Pre-trained Language Modelscited by 133
  20. 2024Granite Code Models: A Family of Open Foundation Models for Code Intelligencecited by 116
  21. 2024EMR-Merging: Tuning-Free High-Performance Model Mergingcited by 108
  22. 2022ReCode: Robustness Evaluation of Code Generation Modelscited by 100
  23. 2023AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Modelscited by 84
  24. 2022Conv-Adapter: Exploring Parameter Efficient Transfer Learning for ConvNetscited by 62
  25. 2023Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learningcited by 53
  26. 2023RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repaircited by 50
  27. 2023Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generationcited by 49
  28. 2024AdaMergeX: Cross-Lingual Transfer with Large Language Models via Adaptive Adapter Mergingcited by 34
  29. 2018Nopol: Automatic Repair of Conditional Statement Bugs in Java Programscited by 32
  30. 2023Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Modelscited by 28
  31. 2019ATOM: Commit Message Generation Based on Abstract Syntax Tree and Hybrid Rankingcited by 25
  32. 2024Astraios: Parameter-Efficient Instruction Tuning Code Large Language Modelscited by 23
  33. 2018FixMiner: Mining Relevant Fix Patterns for Automated Program Repaircited by 22
  34. 2023Towards Efficient Fine-tuning of Pre-trained Code Models: An Experimental Study and Beyondcited by 20
  35. 2024Efficient and Green Large Language Models for Software Engineering: Literature Review, Vision, and the Road Aheadcited by 17
  36. 2023One Adapter for All Programming Languages? Adapter Tuning for Code Search and Summarizationcited by 16
  37. 2022Parameter Efficient Transfer Learning for Various Speech Processing Taskscited by 8
  38. 2024Delving into Parameter-Efficient Fine-Tuning in Code Change Learning: An Empirical Studycited by 8

50 most cited of 54 All 54 (JSON)

Built on by

Not cited yet.