mlsanityData to 7 Oct 2026

FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuning

arXiv 2603.13282 ↗First posted 27 Feb 2026cs.LG cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) with Low-Rank Adaptation (LoRA) has become a standard for privacy-preserving LLM fine-tuning. However, existing personalized methods predominantly operated under a restrictive Flat-Model Assumption: they addressed client-side textitstatistical heterogeneity but treated the model as a monolithic block, ignoring the textitfunctional heterogeneity across LLM layers. We argue that these two statistical (horizontal) and functional (vertical) dimensions, are textitorthogonal in source yet coupled in interaction, implying that the optimal depth of parameter sharing is functionally dependent on client similarity. To address this, we propose FedTreeLoRA, a framework employing tree-structured aggregation for fine-grained, layer-wise alignment. By dynamically constructing an aggregation hierarchy, FedTreeLoRA allows clients to share broad consensus on shallow `trunks' while progressively specializing on deep `branches'. Experiments on NLU and NLG benchmarks demonstrate that FedTreeLoRA significantly outperforms state-of-the-art methods by effectively reconciling generalization and personalization.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 38 of its 50 references that are in the map. First posted on arXiv 27 Feb 2026.

Builds on

38 of 50 references.

References · 38

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  10. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  11. 2019Federated Learning with Personalization Layerscited by 533
  12. 2020An Efficient Framework for Clustered Federated Learningcited by 505
26 more
  1. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 499
  2. 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 322
  3. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 309
  4. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 277
  5. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 184
  6. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 158
  7. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 144
  8. 2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 118
  9. 2024Higher Layers Need More LoRA Expertscited by 101
  10. 2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 100
  11. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 92
  12. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  13. 2024FDLoRA: Personalized Federated Learning of Large Language Model via Dual LoRA Tuningcited by 57
  14. 2024Dual-Personalizing Adapter for Federated Foundation Modelscited by 54
  15. 2024LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinementcited by 37
  16. 2024Taming Cross-Domain Representation Variance in Federated Prototype Learning with Heterogeneous Data Domainscited by 27
  17. 2025FedSWA: Improving Generalization in Federated Learning with Highly Heterogeneous Data via Momentum-Based Stochastic Controlled Weight Averagingcited by 25
  18. 2024FedMM: Federated Multi-Modal Learning with Modality Heterogeneity in Computational Pathologycited by 17
  19. 2023Accelerating Hybrid Federated Learning Convergence under Partial Participationcited by 15
  20. 2025A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learningcited by 15
  21. 2025FedALT: Federated Fine-Tuning through Adaptive Local Training with Rest-of-World LoRAcited by 13
  22. 2025TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by a Hierarchical Gradient-Similarity Treecited by 11
  23. 2025Fed-HeLLo: Efficient Federated Foundation Model Fine-Tuning with Heterogeneous LoRA Allocationcited by 7
  24. 2025Personalized Federated Fine-tuning for Heterogeneous Data: An Automatic Rank Learning Approach via Two-Level LoRAcited by 6
  25. 2025Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuningcited by 5
  26. 2025FedEL: Federated Elastic Learning for Heterogeneous Devicescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026FedWeave: Rethinking the Unit of Specialization in Heterogeneous Federated MoE-LoRAcited by 0
  2. 2026Breaking the Structural Identity: Personalized Federated LoRA Fine-tuning under Rank Heterogeneitycited by 0