mlsanityData to 7 Oct 2026

FedWeave: Rethinking the Unit of Specialization in Heterogeneous Federated MoE-LoRA

arXiv 2607.26618 ↗First posted 29 Jul 2026cs.LG cs.CLcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

Federated PEFT enables LLMs to collaboratively adapt to decentralized private data without sharing raw examples. However, task heterogeneity across clients can cause cross-task interference and gradient conflicts during aggregation. Federated MoE-LoRA addresses this challenge through specialized LoRA experts and conditional routing. Yet existing methods typically specialize at client granularity, implicitly assuming task-coherent clients. Our core insight is that experts need purity, namely pattern-coherent updates that preserve specialization, whereas routers need contrast, namely mixed-task observations that support expert comparison. We propose FedWeave, a framework that adopts asymmetric aggregation, separating expert aggregation from router optimization to meet these two requirements. FedWeave uses unsupervised prototype discovery to form local buckets and align them across clients, enabling prototype-level expert aggregation while retaining mixed-task client trajectories for router training. At inference, FedWeave performs sparse inference with one active expert while preserving nearly all soft-routing performance. Our theoretical analysis explains why asymmetric aggregation is advantageous: it controls expert convergence in stationarity through off-pattern contamination, identifies the consensus error induced by fragmented router trajectories, and bounds sparse-inference risk. On a heterogeneous multi-task benchmark with mainstream LLM backbones, FedWeave consistently outperforms strong baselines, while ablations verify the effectiveness of our design.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 14 of its 46 references that are in the map. First posted on arXiv 29 Jul 2026.

Builds on

14 of 46 references.

References · 14

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  4. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 184
  5. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMscited by 84
  6. 2021Federated Mixture of Expertscited by 27
  7. 2024FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Expertscited by 27
  8. 2020Specialized federated learning using a mixture of expertscited by 23
  9. 2025Personalized Federated Fine-tuning for Heterogeneous Data: An Automatic Rank Learning Approach via Two-Level LoRAcited by 6
  10. 2026Wireless Federated Multi-Task LLM Fine-Tuning via Sparse-and-Orthogonal LoRAcited by 2
  11. 2026FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuningcited by 2
  12. 2026Beyond Factor Aggregation: Gauge-Aware Low-Rank Server Representations for Federated LoRAcited by 2
2 more
  1. 2026SDFLoRA: Selective Decoupled Federated LoRA for Privacy-preserving Fine-tuning with Heterogeneous Clientscited by 1
  2. 2026Task-Centric Personalized Federated Fine-Tuning of Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026MoE Router-Guided Clustering for Heterogeneous Federated Instruction Tuning
  2. 2026UB-SMoE: Universally Balanced Sparse Mixture-of-Experts for Resource-adaptive Federated Fine-tuning of Foundation Models
  3. 2026Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation
  4. 2026Beyond Routing: Decoupling Expert Dispatch and Aggregation in Sparse Mixture-of-Experts
  5. 2026A Persistent State for Auditable Mixture-of-Experts Routing
  6. 2026MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scale
  7. 2026Aggregation Alignment for Federated Learning with Mixture-of-Experts under Data Heterogeneity
  8. 2023Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settings