# Cross-LoRA: A Data-Free LoRA Transfer Framework across Heterogeneous LLMs

arXiv: https://arxiv.org/abs/2508.05232 · HTML: https://mlsanity.com/paper/2508.05232/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 14 references that are in the map.
- First posted on arXiv 7 Aug 2025.

## Builds on (most cited first, 12 of 12)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [DoRA: Weight-Decomposed Low-Rank Adaptation](https://mlsanity.com/paper/2402.09353/) (2024)
- [LoRA+: Efficient Low Rank Adaptation of Large Models](https://mlsanity.com/paper/2402.12354/) (2024)
- [Towards Understanding Knowledge Distillation](https://mlsanity.com/paper/2105.13093/) (2021)
- [Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights](https://mlsanity.com/paper/2506.16406/) (2025)
- [textitTrans-LoRA: towards data-free Transferable Parameter Efficient Finetuning](https://mlsanity.com/paper/2405.17258/) (2024)
- [LoRA-X: Bridging Foundation Models with Training-Free Cross-Model Adaptation](https://mlsanity.com/paper/2501.16559/) (2025)
- [LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades](https://mlsanity.com/paper/2505.13515/) (2025)
- [Zero-Shot Adaptation of Parameter-Efficient Fine-Tuning in Diffusion Models](https://mlsanity.com/paper/2506.04244/) (2025)

## Built on by (most cited first, 10 of 10)

- [A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthiness](https://mlsanity.com/paper/2510.13890/) (2025)
- [Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigms](https://mlsanity.com/paper/2608.27409/) (2026)
- [Frozen Memory Is Not Enough: Rethinking External Memory as Extraction](https://mlsanity.com/paper/2608.17050/) (2026)
- [LLM Benchmark Datasets Should Be Contamination-Resistant](https://mlsanity.com/paper/2605.19999/) (2026)
- [Parametric Memory Decoding for Zero-Shot Routing in LoRA-Based External Parametric Memory](https://mlsanity.com/paper/2607.04118/) (2026)
- [TuneShift-KD: Knowledge Distillation and Transfer for Fine-tuned Models](https://mlsanity.com/paper/2603.24518/) (2026)
- [Activation Manifold Projection: Liberating Task-Specific Behaviors from LLM Architectures](https://mlsanity.com/paper/2510.17902/) (2025)
- [Context Recycling for Long-Horizon LLM Inference](https://mlsanity.com/paper/2606.26105/) (2026)
- [UpgradeBench: A Decision-Centric Benchmark for Upgrading Fine-Tuned LLM Specialists](https://mlsanity.com/paper/2608.20918/) (2026)
- [Platonic Task Arithmetic](https://mlsanity.com/paper/2610.00929/) (2026)

## Abstract (arXiv)

Traditional parameter-efficient fine-tuning (PEFT) methods such as LoRA are tightly coupled with the base model architecture, which constrains their applicability across heterogeneous pretrained large language models (LLMs). To address this limitation, we introduce Cross-LoRA, a data-free framework for transferring LoRA modules between diverse base models without requiring additional training data. Cross-LoRA consists of two key components: (a) LoRA-Align, which performs subspace alignment between source and target base models through rank-truncated singular value decomposition (SVD) and Frobenius-optimal linear transformation, ensuring compatibility under dimension mismatch; and (b) LoRA-Shift, which applies the aligned subspaces to project source LoRA weight updates into the target model parameter space. Both components are data-free, training-free, and enable lightweight adaptation on a commodity GPU in 20 minutes. Experiments on ARCs, OBOA and HellaSwag show that Cross-LoRA achieves relative gains of up to 5.26% over base models. Across other commonsense reasoning benchmarks, Cross-LoRA maintains performance comparable to that of directly trained LoRA adapters.