# Accurate and Efficient Low-Rank Model Merging in Core Space

arXiv: https://arxiv.org/abs/2509.17786 · HTML: https://mlsanity.com/paper/2509.17786/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 55 references that are in the map.
- First posted on arXiv 22 Sep 2025.

## Builds on (most cited first, 30 of 40)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [Remote Sensing Image Scene Classification: Benchmark and State of the Art](https://mlsanity.com/paper/1703.00121/) (2017)
- [DoRA: Weight-Decomposed Low-Rank Adaptation](https://mlsanity.com/paper/2402.09353/) (2024)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021)
- [Dataless Knowledge Fusion by Merging Weights of Language Models](https://mlsanity.com/paper/2212.09849/) (2022)
- [AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2303.10512/) (2023)
- [Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities](https://mlsanity.com/paper/2408.07666/) (2024)
- [AdaMerging: Adaptive Model Merging for Multi-Task Learning](https://mlsanity.com/paper/2310.02575/) (2023)
- [Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models](https://mlsanity.com/paper/2305.12827/) (2023)
- [Evolutionary Optimization of Model Merging Recipes](https://mlsanity.com/paper/2403.13187/) (2024)
- [Task Singular Vectors: Reducing Task Interference in Model Merging](https://mlsanity.com/paper/2412.00081/) (2024)
- [Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Masks](https://mlsanity.com/paper/2312.06795/) (2023)
- [Localizing Task Information for Improved Model Merging and Compression](https://mlsanity.com/paper/2405.07813/) (2024)
- [Model merging with SVD to tie the Knots](https://mlsanity.com/paper/2410.19735/) (2024)

## Built on by (most cited first, 29 of 29)

- [Transporting Task Vectors across Different Architectures without Training](https://mlsanity.com/paper/2602.12952/) (2026)
- [ACE-Merging: Data-Free Model Merging with Adaptive Covariance Estimation](https://mlsanity.com/paper/2603.02945/) (2026)
- [Gradient-Sign Masking for Task Vector Transport Across Pre-Trained Models](https://mlsanity.com/paper/2510.09658/) (2025)
- [CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging](https://mlsanity.com/paper/2603.00573/) (2026)
- [Label-Free Cross-Task LoRA Merging with Null-Space Compression](https://mlsanity.com/paper/2603.26317/) (2026)
- [Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvature](https://mlsanity.com/paper/2602.17385/) (2026)
- [TRACE: Capability-Targeted Agentic Training](https://mlsanity.com/paper/2604.05336/) (2026)
- [Compress then Merge: From Multiple LoRAs into One Low-Rank Adapter](https://mlsanity.com/paper/2606.03723/) (2026)
- [Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Merging](https://mlsanity.com/paper/2512.01461/) (2025)
- [Preference-Aligned LoRA Merging: Preserving Subspace Coverage and Addressing Directional Anisotropy](https://mlsanity.com/paper/2603.26299/) (2026)
- [Making Models Unmergeable via Scaling-Sensitive Loss Landscape](https://mlsanity.com/paper/2601.21898/) (2026)
- [LoRA-Squeeze: Simple and Effective Post-Tuning and In-Tuning Compression of LoRA Modules](https://mlsanity.com/paper/2602.10993/) (2026)
- [Generalizing the Geometry of Model Merging Through Frechet Averages](https://mlsanity.com/paper/2604.27155/) (2026)
- [Distilling Linearized Behavior into Non-Linear Fine-Tuning for Effective Task Arithmetic](https://mlsanity.com/paper/2605.18993/) (2026)
- [Beyond Classification: Dynamic Adapter Routing for Continual Multimodal Retrieval](https://mlsanity.com/paper/2605.31229/) (2026)
- [SSR-Merge: Subspace Signal Routing for Training-Free LoRA Merging in Diffusion Models](https://mlsanity.com/paper/2606.10617/) (2026)
- [Unified Prediction and Planning via Conflict-Aware Disjoint Parameter Training](https://mlsanity.com/paper/2607.19971/) (2026)
- [SSAM: Singular Subspace Alignment for Merging Multimodal Large Language Models](https://mlsanity.com/paper/2603.21584/) (2026)
- [Zero-Shot Synthetic-to-Real Handwritten Text Recognition via Task Analogies](https://mlsanity.com/paper/2604.09713/) (2026)
- [Crowded in B-Space: Calibrating Shared Directions for LoRA Merging](https://mlsanity.com/paper/2604.16826/) (2026)
- [Evolutionary Negative Module Pruning for Better LoRA Merging](https://mlsanity.com/paper/2604.17753/) (2026)
- [Domain Arithmetic: One-Shot VLA Adaptation under Environmental Shifts](https://mlsanity.com/paper/2607.00666/) (2026)
- [Model Merging for Medical LVLMs: A Benchmark and a Winner-Take-All Approach](https://mlsanity.com/paper/2607.15661/) (2026)
- [CT-Merging: Consensus Directions and Task-Specific Scaling for LoRA Adapter Merging](https://mlsanity.com/paper/2607.20561/) (2026)
- [Rethinking Expert Training for Model Merging with Prompt Learning](https://mlsanity.com/paper/2607.24465/) (2026)
- [FraQ: Efficient Coordinate-Space Recompression for Federated Low-Rank Adaptation](https://mlsanity.com/paper/2608.03605/) (2026)
- [CoRe-MoE: Compact Reusable MoE for Continual Multimodal Instruction Tuning](https://mlsanity.com/paper/2608.27867/) (2026)
- [Train Together or Merge Later? Unifying VLA Experts via a Shared Action Interface](https://mlsanity.com/paper/2609.33125/) (2026)
- [PGSB: Pretrained-Guided Shared Basis for LoRA Model Merging](https://mlsanity.com/paper/2610.14558/) (2026)

## Abstract (arXiv)

In this paper, we address the challenges associated with merging low-rank adaptations of large neural networks. With the rise of parameter-efficient adaptation techniques, such as Low-Rank Adaptation (LoRA), model fine-tuning has become more accessible. While fine-tuning models with LoRA is highly efficient, existing merging methods often sacrifice this efficiency by merging fully-sized weight matrices. We propose the Core Space merging framework, which enables the merging of LoRA-adapted models within a common alignment basis, thereby preserving the efficiency of low-rank adaptation while substantially improving accuracy across tasks. We further provide a formal proof that projection into Core Space ensures no loss of information and provide a complexity analysis showing the efficiency gains. Extensive empirical results demonstrate that Core Space significantly improves existing merging techniques and achieves state-of-the-art results on both vision and language tasks while utilizing a fraction of the computational resources. Codebase is available at https://github.com/apanariello4/core-space-merging.