mlsanityData to 7 Oct 2026

Can LoRA Fusion Support Cross-Domain Tasks in Cloud-Edge Collaboration?

arXiv 2605.23913 ↗First posted 9 Apr 2026cs.DC cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Cloud-hosted large language models (LLMs) commonly rely on LoRA for domain adaptation, yet domain data are distributed across multiple edge devices and cannot be uploaded due to privacy constraints. This raises a fundamental question: how can knowledge from multiple private edges be integrated into a cloud LLM for cross-domain problem solving? A natural solution is to train LoRA adapters locally and fuse them in the cloud; however, existing pipelines rely on unrealistic assumptions that edge devices can host cloud-scale LLMs and are evaluated mainly on single-domain tasks. To address these limitations, we propose a prune-train-recover framework that enables local LoRA training on pruned models and privacy-preserving cloud integration. We further introduce MMLU-CD, a cross-domain benchmark that composes multiple domain samples into a single instance, enabling explicit evaluation of cross-domain problem solving. This allows us to ask a concrete question: Can existing LoRA fusion methods support cross-domain tasks in cloud-edge collaboration? Our empirical answer is negative. Existing LoRA fusion methods perform poorly on MMLU-CD, often underperforming the base LLM, revealing their inability to support cross-domain problem solving. We attribute this failure to parameter conflicts among LoRA adapters and propose a simple conflict-resolution module, LoRA-CR, which mitigates conflicting updates and improves LoRA fusion performance by up to 3.8%. These results identify conflict mitigation as a critical yet largely overlooked factor in cloud-edge LoRA fusion, warranting further investigation in future research.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 42 of its 57 references that are in the map. First posted on arXiv 9 Apr 2026.

Builds on

42 of 57 references.

References · 42

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  7. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  8. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  9. 2022Editing Models with Task Arithmeticcited by 1,076
  10. 2020In Search of Lost Domain Generalizationcited by 1,042
  11. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  12. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
30 more
  1. 2022Continual Test-Time Domain Adaptationcited by 520
  2. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  3. 2021Gradient Projection Memory for Continual Learningcited by 338
  4. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 317
  5. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 248
  6. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  7. 2025Dynamic Early Exit in Reasoning Modelscited by 217
  8. 2023DepGraph: Towards Any Structural Pruningcited by 202
  9. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 158
  10. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 146
  11. 2024A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthinesscited by 125
  12. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 124
  13. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  14. 2025Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectorscited by 59
  15. 2024FDLoRA: Personalized Federated Learning of Large Language Model via Dual LoRA Tuningcited by 57
  16. 2024Dual-Personalizing Adapter for Federated Foundation Modelscited by 54
  17. 2024Revisiting Weight Averaging for Model Mergingcited by 24
  18. 2023Low-Parameter Federated Learning with Large Language Modelscited by 22
  19. 2025CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Mergingcited by 22
  20. 2023An Interdisciplinary Outlook on Large Language Models for Scientific Researchcited by 15
  21. 2024Safely Learning with Private Data: A Federated Learning Framework for Large Language Modelcited by 11
  22. 2025NeedleInATable: Exploring Long-Context Capability of Large Language Models towards Long-Structured Tablescited by 8
  23. 2025Mediator: Memory-efficient LLM Merging with Less Parameter Conflicts and Uncertainty Based Routingcited by 7
  24. 2023Learning to Generalize for Cross-domain QAcited by 6
  25. 2025Train Small, Infer Large: Memory-Efficient LoRA Training for Large Language Modelscited by 5
  26. 2025A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthinesscited by 5
  27. 2024Domain Generalization via Causal Adjustment for Cross-Domain Sentiment Analysiscited by 3
  28. 2025BeamLoRA: Beam-Constraint Low-Rank Adaptationcited by 3
  29. 2025Blink: Dynamic Visual Token Resolution for Enhanced Multimodal Understandingcited by 2
  30. 2025CBP-Tuning: Efficient Local Customization for Black-box Large Language Modelscited by 1

Built on by

Not cited yet.