# Adaptive LoRA Merge with Parameter Pruning for Low-Resource Generation

arXiv: https://arxiv.org/abs/2505.24174 · HTML: https://mlsanity.com/paper/2505.24174/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 37 references that are in the map.
- First posted on arXiv 30 May 2025.

## Builds on (most cited first, 21 of 21)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [Importance Estimation for Neural Network Pruning](https://mlsanity.com/paper/1906.10771/) (2019)
- [LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition](https://mlsanity.com/paper/2307.13269/) (2023)
- [XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages](https://mlsanity.com/paper/2106.13822/) (2021)
- [TLDR: Extreme Summarization of Scientific Documents](https://mlsanity.com/paper/2004.15011/) (2020)
- [Large Language Models Meet NLP: A Survey](https://mlsanity.com/paper/2405.12819/) (2024)
- [The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts](https://mlsanity.com/paper/2401.13136/) (2024)
- [PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importance](https://mlsanity.com/paper/2206.12562/) (2022)
- [LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wild](https://mlsanity.com/paper/2402.09997/) (2024)
- [Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization](https://mlsanity.com/paper/2105.12002/) (2021)
- [Knowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMs](https://mlsanity.com/paper/2308.09895/) (2023)
- [LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks](https://mlsanity.com/paper/2402.11455/) (2024)
- [ConStruct-VL: Data-Free Continual Structured VL Concepts Learning](https://mlsanity.com/paper/2211.09790/) (2022)
- [Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuning](https://mlsanity.com/paper/2411.18571/) (2024)
- [Fine-Tuning Medical Language Models for Enhanced Long-Contextual Understanding and Domain Expertise](https://mlsanity.com/paper/2407.11536/) (2024)
- [To be or not to be? an exploration of continuously controllable prompt engineering](https://mlsanity.com/paper/2311.09773/) (2023)

## Built on by (most cited first, 5 of 5)

- [HiLoRA: Adaptive Hierarchical LoRA Routing for Training-Free Domain Generalization](https://mlsanity.com/paper/2510.12266/) (2025)
- [Towards Efficient Multimodal Unified Reasoning Model via Model Merging](https://mlsanity.com/paper/2510.08987/) (2025)
- [From Parameters to Behaviors: A Survey of Model Fusion for Large Language Models](https://mlsanity.com/paper/2609.19553/) (2026)
- [Black-box Model Merging for Language-Model-as-a-Service with Massive Model Repositories](https://mlsanity.com/paper/2509.12951/) (2025)
- [GRASP LoRA: GRPO Guided Adapter Sparsity Policy for Cross Lingual Transfer](https://mlsanity.com/paper/2601.06702/) (2026)

## Abstract (arXiv)

This study proposes a simple yet effective LoRA merge method to achieve LLM adaptation for low-resource language generation tasks. The LoRA merge technique, which integrates multiple LoRA modules trained on different tasks, has gained attention as an effective and efficient approach for adapting LLMs to target tasks. However, previous methods are limited in adaptability as they keep the LoRA parameters frozen. Additionally, the low-resource problem has been out of their scope. We propose a LoRA merge method that updates and prunes LoRA parameters through fine-tuning with minimal target task data, which allows finer-grained adjustments of LoRA parameters and enhancement of task adaptability. Extensive experiments have been conducted taking summarization as a benchmark task. Our datasets cover various domains and multiple languages of English and Japanese. The results confirm that the proposed method achieves significant and consistent improvements in task adaptability over the previous methods.