# Wireless Federated Multi-Task LLM Fine-Tuning via Sparse-and-Orthogonal LoRA

arXiv: https://arxiv.org/abs/2602.20492 · HTML: https://mlsanity.com/paper/2602.20492/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Federated mixture-of-experts, one of 12 directions in Mixture-of-experts.
- Builds on 22 of its 31 references that are in the map.
- First posted on arXiv 24 Feb 2026.

## Placed in

- Mixture-of-experts / [Federated mixture-of-experts](https://mlsanity.com/moe/federated-mixture-of-experts/)

## Builds on (most cited first, 22 of 22)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Orthogonal Gradient Descent for Continual Learning](https://mlsanity.com/paper/1910.07104/) (2019)
- [Orthogonal Subspace Learning for Language Model Continual Learning](https://mlsanity.com/paper/2310.14152/) (2023)
- [A Joint Learning and Communications Framework for Federated Learning over Wireless Networks](https://mlsanity.com/paper/1909.07972/) (2019)
- [Wireless Communications for Collaborative Federated Learning](https://mlsanity.com/paper/2006.02499/) (2020)
- [FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts](https://mlsanity.com/paper/2510.08396/) (2025)
- [Large Language Models for Networking: Applications, Enabling Techniques, and Challenges](https://mlsanity.com/paper/2311.17474/) (2023)
- [LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation](https://mlsanity.com/paper/2504.07448/) (2025)
- [Is Parameter Collision Hindering Continual Learning in LLMs?](https://mlsanity.com/paper/2410.10179/) (2024)
- [OMoE: Diversifying Mixture of Low-Rank Adaptation by Orthogonal Finetuning](https://mlsanity.com/paper/2501.10062/) (2025)
- [FedQuad: Adaptive Layer-wise LoRA Deployment and Activation Quantization for Federated Fine-Tuning](https://mlsanity.com/paper/2506.01001/) (2025)
- [dFLMoE: Decentralized Federated Learning via Mixture of Experts for Medical Data Analysis](https://mlsanity.com/paper/2503.10412/) (2025)
- [Token-Level LLM Collaboration via FusionRoute](https://mlsanity.com/paper/2601.05106/) (2026)
- [Federated Fine-Tuning of LLMs: Framework Comparison and Research Directions](https://mlsanity.com/paper/2501.04436/) (2025)
- [Fed-HeLLo: Efficient Federated Foundation Model Fine-Tuning with Heterogeneous LoRA Allocation](https://mlsanity.com/paper/2506.12213/) (2025)
- [FFT-MoE: Efficient Federated Fine-Tuning for Foundation Models via Large-scale Sparse MoE under Heterogeneous Edge](https://mlsanity.com/paper/2508.18663/) (2025)
- [MoORE: SVD-based Model MoE-ization for Conflict- and Oblivion-Resistant Multi-Task Adaptation](https://mlsanity.com/paper/2506.14436/) (2025)
- [ADF-LoRA: Alternating Low-Rank Aggregation for Decentralized Federated Fine-Tuning](https://mlsanity.com/paper/2511.18291/) (2025)
- [ThanoRA: Task Heterogeneity-Aware Multi-Task Low-Rank Adaptation](https://mlsanity.com/paper/2505.18640/) (2025)
- [Two Stage Wireless Federated LoRA Fine-Tuning with Sparsified Orthogonal Updates](https://mlsanity.com/paper/2505.00333/) (2025)
- [Learning to Collaborate: An Orchestrated-Decentralized Framework for Peer-to-Peer LLM Federation](https://mlsanity.com/paper/2601.17133/) (2026)

## Built on by (most cited first, 2 of 2)

- [Priority-Aware Learning-Unlearning Correction for Dynamic Decentralized LoRA Fine-Tuning](https://mlsanity.com/paper/2606.22878/) (2026)
- [FedWeave: Rethinking the Unit of Specialization in Heterogeneous Federated MoE-LoRA](https://mlsanity.com/paper/2607.26618/) (2026)

## Abstract (arXiv)

Decentralized federated learning (DFL) based on low-rank adaptation (LoRA) enables mobile devices with multi-task datasets to collaboratively fine-tune a large language model (LLM) by exchanging locally updated parameters with a subset of neighboring devices via wireless connections for knowledge integration.However, directly aggregating parameters fine-tuned on heterogeneous datasets induces three primary issues across the DFL life-cycle: (i) textitcatastrophic knowledge forgetting during fine-tuning process, arising from conflicting update directions caused by data heterogeneity; (ii) textitinefficient communication and convergence during model aggregation process, due to bandwidth-intensive redundant model transmissions; and (iii) textitmulti-task knowledge interference during inference process, resulting from incompatible knowledge representations coexistence during inference. To address these issues in a fully decentralized scenario, we first propose a sparse-and-orthogonal LoRA that ensures orthogonality between model updates to eliminate direction conflicts during fine-tuning.Then, we analyze how device connection topology affects multi-task performance, prompting a cluster-based topology design during aggregation.Finally, we propose an implicit mixture of experts (MoE) mechanism to avoid the coexistence of incompatible knowledge during inference. Simulation results demonstrate that the proposed approach effectively reduces communication resource consumption by up to 73% and enhances average performance by 5% compared with the traditional LoRA method.