# Elastic Mixture of Rank-Wise Experts for Knowledge Reuse in Federated Fine-Tuning

arXiv: https://arxiv.org/abs/2512.00902 · HTML: https://mlsanity.com/paper/2512.00902/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Nov 2025.

## Built on by (most cited first, 6 of 6)

- [TSEmbed: Unlocking Task Scaling in Universal Multimodal Embeddings](https://mlsanity.com/paper/2603.04772/) (2026)
- [When Agents "Misremember" Collectively: Exploring the Mandela Effect in LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2602.00428/) (2026)
- ["I See What You Did There": Can Large Vision-Language Models Understand Multimodal Puns?](https://mlsanity.com/paper/2604.05930/) (2026)
- [Spotlight and Shadow: Attention-Guided Dual-Anchor Introspective Decoding for MLLM Hallucination Mitigation](https://mlsanity.com/paper/2604.10071/) (2026)
- [FBLayout: Optimizing Memory Layout for Efficient LLM Finetuning on Mobile GPUs](https://mlsanity.com/paper/2607.21624/) (2026)
- [How Many Ranks Does a Token Need? A Noise-Floor Squelch for Rank-One LoRA Experts](https://mlsanity.com/paper/2610.19033/) (2026)

## Abstract (arXiv)

Federated fine-tuning offers a promising solution for adapting Large Language Models (LLMs) to downstream tasks while safeguarding data privacy. However, its high computational and communication demands hinder its deployment on resource-constrained devices. In this paper, we propose SmartFed, a resource-efficient federated fine-tuning framework. SmartFed intelligently reuses knowledge embedded in existing LoRA modules, eliminating the need for expensive training from scratch when adapting LLMs to new tasks. To effectively exploit this knowledge and ensure scalability, we introduce the Mixture of Rank-Wise Experts (MoRE). MoRE decomposes LoRA modules into fine-grained rank-level experts. These experts are selectively activated and combined based on input semantics and resource budgets. Moreover, to optimize resource utilization, we present the Elastic Expert Quota Allocation (EEQA). EEQA adaptively allocates expert capacity across parameter matrices based on their contribution to model performance, focusing computing resources on the critical experts. Extensive evaluations across multiple benchmarks demonstrate that SmartFed significantly outperforms existing methods in model performance and training efficiency.