mlsanityData to 7 Oct 2026

Wireless Federated Multi-Task LLM Fine-Tuning via Sparse-and-Orthogonal LoRA

arXiv 2602.20492 ↗First posted 24 Feb 2026cs.LG cs.AIcited by 2

On the map: Mixture-of-experts → Federated mixture-of-experts

Abstract (arXiv)

Decentralized federated learning (DFL) based on low-rank adaptation (LoRA) enables mobile devices with multi-task datasets to collaboratively fine-tune a large language model (LLM) by exchanging locally updated parameters with a subset of neighboring devices via wireless connections for knowledge integration.However, directly aggregating parameters fine-tuned on heterogeneous datasets induces three primary issues across the DFL life-cycle: (i) textitcatastrophic knowledge forgetting during fine-tuning process, arising from conflicting update directions caused by data heterogeneity; (ii) textitinefficient communication and convergence during model aggregation process, due to bandwidth-intensive redundant model transmissions; and (iii) textitmulti-task knowledge interference during inference process, resulting from incompatible knowledge representations coexistence during inference. To address these issues in a fully decentralized scenario, we first propose a sparse-and-orthogonal LoRA that ensures orthogonality between model updates to eliminate direction conflicts during fine-tuning.Then, we analyze how device connection topology affects multi-task performance, prompting a cluster-based topology design during aggregation.Finally, we propose an implicit mixture of experts (MoE) mechanism to avoid the coexistence of incompatible knowledge during inference. Simulation results demonstrate that the proposed approach effectively reduces communication resource consumption by up to 73% and enhances average performance by 5% compared with the traditional LoRA method.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Federated mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 22 of its 31 references that are in the map. First posted on arXiv 24 Feb 2026.

Builds on

22 of 31 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025FFT-MoE: Efficient Federated Fine-Tuning for Foundation Models via Large-scale Sparse MoE under Heterogeneous Edge26 Aug
  2. 2025dFLMoE: Decentralized Federated Learning via Mixture of Experts for Medical Data Analysis13 Mar

Other references · 20

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2019Orthogonal Gradient Descent for Continual Learningcited by 361
  4. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 296
  5. 2019A Joint Learning and Communications Framework for Federated Learning over Wireless Networkscited by 262
  6. 2020Wireless Communications for Collaborative Federated Learningcited by 47
  7. 2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 34
  8. 2023Large Language Models for Networking: Applications, Enabling Techniques, and Challengescited by 28
  9. 2025LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptationcited by 18
  10. 2024Is Parameter Collision Hindering Continual Learning in LLMs?cited by 15
  11. 2025OMoE: Diversifying Mixture of Low-Rank Adaptation by Orthogonal Finetuningcited by 14
  12. 2025FedQuad: Adaptive Layer-wise LoRA Deployment and Activation Quantization for Federated Fine-Tuningcited by 10
8 more
  1. 2026Token-Level LLM Collaboration via FusionRoutecited by 9
  2. 2025Federated Fine-Tuning of LLMs: Framework Comparison and Research Directionscited by 8
  3. 2025Fed-HeLLo: Efficient Federated Foundation Model Fine-Tuning with Heterogeneous LoRA Allocationcited by 7
  4. 2025MoORE: SVD-based Model MoE-ization for Conflict- and Oblivion-Resistant Multi-Task Adaptationcited by 4
  5. 2025ADF-LoRA: Alternating Low-Rank Aggregation for Decentralized Federated Fine-Tuningcited by 4
  6. 2025ThanoRA: Task Heterogeneity-Aware Multi-Task Low-Rank Adaptationcited by 3
  7. 2025Two Stage Wireless Federated LoRA Fine-Tuning with Sparsified Orthogonal Updatescited by 2
  8. 2026Learning to Collaborate: An Orchestrated-Decentralized Framework for Peer-to-Peer LLM Federationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Priority-Aware Learning-Unlearning Correction for Dynamic Decentralized LoRA Fine-Tuningcited by 0
  2. 2026FedWeave: Rethinking the Unit of Specialization in Heterogeneous Federated MoE-LoRAcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation
  2. 2024Personalized Wireless Federated Learning for Large Language Models
  3. 2026Federated LoRA Fine-Tuning for LLMs via Collaborative Alignment
  4. 2025Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning
  5. 2026FraQ: Efficient Coordinate-Space Recompression for Federated Low-Rank Adaptation
  6. 2026FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignment
  7. 2025Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAs
  8. 2025Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning