Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models
Not on the map. Why?
Abstract (arXiv)
Foundation models (FMs) adapt well to specific domains or tasks with fine-tuning, and federated learning (FL) enables the potential for privacy-preserving fine-tuning of the FMs with on-device local data. For federated fine-tuning of FMs, we consider the FMs with small to medium parameter sizes of single digit billion at maximum, referred to as on-device FMs (ODFMs) that can be deployed on devices for inference but can only be fine-tuned with parameter efficient methods. In our work, we tackle the data and system heterogeneity problem of federated fine-tuning of ODFMs by proposing a novel method using heterogeneous low-rank approximations (LoRAs), namely HetLoRA. First, we show that the naive approach of using homogeneous LoRA ranks across devices face a trade-off between overfitting and slow convergence, and thus propose HetLoRA, which allows heterogeneous ranks across client devices and efficiently aggregates and distributes these heterogeneous LoRA modules. By applying rank self-pruning locally and sparsity-weighted aggregation at the server, HetLoRA combines the advantages of high and low-rank LoRAs, which achieves improved convergence speed and final performance compared to homogeneous LoRA. Furthermore, HetLoRA offers enhanced computation efficiency compared to full fine-tuning, making it suitable for federated fine-tuning across heterogeneous devices.
Key facts. Cited by 114 papers in the map (data through 7 Oct 2026). Builds on 34 of its 45 references that are in the map. First posted on arXiv 12 Jan 2024.
Builds on
34 of 45 references.
References · 34
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2023GPT-4 Technical Reportcited by 18,132
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
- 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
22 more
- 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
- 2018Federated Optimization in Heterogeneous Networkscited by 2,971
- 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
- 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
- 2023PaLM 2 Technical Reportcited by 1,154
- 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
- 2018Adaptive Federated Learning in Resource Constrained Edge Computing Systemscited by 408
- 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 364
- 2021A Field Guide to Federated Optimizationcited by 312
- 2021Beyond Goldfish Memory: Long-Term Open-Domain Conversationcited by 266
- 2016Practical Secure Aggregation for Federated Learning on User-Held Datacited by 178
- 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 133
- 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 92
- 2019What Would Elsa Do? Freezing Layers During Transformer Fine-Tuningcited by 87
- 2021Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuningcited by 82
- 2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 74
- 2023Federated Foundation Models: Privacy-Preserving and Collaborative Learning for Large Modelscited by 49
- 2022FedTune: A Deep Dive into Efficient Federated Fine-Tuning with Pre-trained Transformerscited by 33
- 2022FiT: Parameter Efficient Few-shot Transfer Learning for Personalized and Federated Image Classificationcited by 26
- 2022lo-fi: distributed fine-tuning without communicationcited by 19
- 2022Federated Multilingual Models for Medical Transcript Analysiscited by 11
- 2023Federated Learning of Medical Concepts Embedding using BEHRTcited by 11
Built on by
Citing papers per year: 2024: 29; 2025: 48; 2026: 37.
114 papers.
Sort:
By area:
- 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 144
- 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
- 2024A Survey on LoRA of Large Language Modelscited by 67
- 2024LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinementcited by 37
- 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
- 2024Advances and Open Challenges in Federated Foundation Modelscited by 33
- 2025HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 30
- 2024FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Modelscited by 26
- 2024Federated LoRA with Sparse Communicationcited by 26
- 2024Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clientscited by 22
- 2024An overview of domain-specific foundation model: key technologies, applications and challengescited by 19
- 2024FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Foundation Modelscited by 18
38 more
- 2024Federated Learning in Practice: Reflections and Projectionscited by 15
- 2025A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learningcited by 15
- 2024Synergizing Foundation Models and Federated Learning: A Surveycited by 14
- 2025FedALT: Federated Fine-Tuning through Adaptive Local Training with Rest-of-World LoRAcited by 13
- 2025LoFT: Low-Rank Adaptation That Behaves Like Full Fine-Tuningcited by 13
- 2024Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Modelscited by 12
- 2025Decentralized Low-Rank Fine-Tuning of Large Language Modelscited by 12
- 2024Towards Federated Low-Rank Adaptation of Language Models with Rank Heterogeneitycited by 10
- 2025Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuningcited by 10
- 2025Federated Low-Rank Adaptation for Foundation Models: A Surveycited by 9
- 2025Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuningcited by 9
- 2024RBLA: Rank-Based-LoRA-Aggregation for Fine-tuning Heterogeneous Models in FLaaScited by 8
- 2025Fed-HeLLo: Efficient Federated Foundation Model Fine-Tuning with Heterogeneous LoRA Allocationcited by 7
- 2024Communication-Efficient Federated Low-Rank Update Algorithm and its Connection to Implicit Regularizationcited by 6
- 2024On-Device Collaborative Language Modeling via a Mixture of Generalists and Specialistscited by 6
- 2024Exploring Gradient Subspaces: Addressing and Overcoming LoRA's Limitations in Federated Fine-Tuning of Large Language Modelscited by 6
- 2024SPD-CFL: Stepwise Parameter Dropout for Efficient Continual Federated Learningcited by 5
- 2024Adaptive Parameter-Efficient Federated Fine-Tuning on Heterogeneous Devicescited by 5
- 2026FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignmentcited by 5
- 2024DEeR: Deviation Eliminating and Noise Regulating for Privacy-preserving Federated Low-rank Adaptationcited by 4
- 2025Adaptive Rank Allocation for Federated Parameter-Efficient Fine-Tuning of Language Modelscited by 4
- 2025Resource-Efficient Federated Fine-Tuning Large Language Models for Heterogeneous Datacited by 4
- 2025FedMerge: Federated Personalization via Model Mergingcited by 4
- 2025FLoRIST: Singular Value Thresholding for Efficient and Accurate Federated Fine-Tuning of Large Language Modelscited by 4
- 2025Learning Like Humans: Resource-Efficient Federated Fine-Tuning through Cognitive Developmental Stagescited by 4
- 2026Rethinking LoRA for Privacy-Preserving Federated Learning in Large Modelscited by 4
- 2024FLoCoRA: Federated learning compression with low-rank adaptationcited by 3
- 2024MobiZO: Enabling Efficient LLM Fine-Tuning at the Edge via Inference Enginescited by 3
- 2024Fed-pilot: Optimizing LoRA Allocation for Efficient Federated Fine-Tuning with Heterogeneous Clientscited by 3
- 2025Many-Task Federated Fine-Tuning via Unified Task Vectorscited by 3
- 2025FedHL: Federated Learning for Heterogeneous Low-Rank Adaptation via Unbiased Aggregationcited by 3
- 2025Converge Faster, Talk Less: Hessian-Informed Federated Zeroth-Order Optimizationcited by 3
- 2026FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuningcited by 3
- 2024Leveraging Unstructured Text Data for Federated Instruction Tuning of Large Language Modelscited by 2
- 2024HAFLQ: Heterogeneous Adaptive Federated LoRA Fine-tuned LLM with Quantizationcited by 2
- 2024HeteroTune: Efficient Federated Learning for Large Heterogeneous Modelscited by 2
- 2025Federated Sketching LoRA: A Flexible Framework for Heterogeneous Collaborative Fine-Tuning of LLMscited by 2
- 2025CLLoRA: An Approach to Measure the Effects of the Context Length for LLM Fine-Tuningcited by 2
50 most cited of 114 All 114 (JSON)
- 2026Federated Agent Optimization1 Oct 2026
- 2026Beyond Uniform Compression: Budgeted Transmission Allocation for Extreme Federated Learning30 Sep 2026
- 2026CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation27 Sep 2026
- 2026Perplexity Predicts Protection: Choosing Pretrained Backbones for Worst-Client Fairness in Federated Parameter-Efficient Fine-Tuning20 Sep 2026
- 2026Co-VLA: Consensus-based Federated Training for Vision-Language-Action Models17 Sep 2026
- 2026CIDERS: Cloud-Edge LLM Collaborative Learning via Accelerating Personalized Bilevel Optimization14 Sep 2026
- 2026Breaking the Structural Identity: Personalized Federated LoRA Fine-tuning under Rank Heterogeneity1 Sep 2026
- 2026Unveiling the Depth-Performance Dilemma in Split-Federated Fine-tuning of LLMs23 Aug 2026
- 2026FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA15 Aug 2026
- 2026SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Ranks10 Aug 2026
- 2026Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approach10 Aug 2026
- 2026MineGrad: Gradient Inversion Attacks on LoRA Fine-Tuning2 Aug 2026
38 more
- 2026Federated Foundation Models Fine-Tuning with Heterogeneous Compressed Clients31 Jul 2026
- 2026Spectral Transformation for Layer-wise Global Rank Discovery in Federated LoRA for Vision Transformers23 Jul 2026
- 2026Three-Pronged Spectral Control for Federated Parameter Efficient Fine Tuning23 Jul 2026
- 2026Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learning15 Jul 2026
- 2026PreLort: Prefix-Nested LoRA for Federated Fine-Tuning under Rank Heterogeneity14 Jun 2026
- 2026Shift-Dependent Asymmetry: Orthogonal Inverse Low-Rank Adaptation for Federated Medical Segmentation7 Jun 2026
- 2026DECA: Decentralizing Block-Wise Adam for Efficient LLM Full-Parameter Fine-Tuning on Non-IID Data2 Jun 2026
- 2026Silent Failures in Federated Personalization of Foundation Models31 May 2026
- 2026UB-SMoE: Universally Balanced Sparse Mixture-of-Experts for Resource-adaptive Federated Fine-tuning of Foundation Models15 May 2026
- 2026Beyond Parameter Aggregation: Semantic Consensus for Federated Fine-Tuning of LLMs12 May 2026
- 2026DP-LAC: Lightweight Adaptive Clipping for Differentially Private Federated Fine-tuning of Language Models11 May 2026
- 2026EdgeFlowerTune: Evaluating Federated LLM Fine-Tuning Under Realistic Edge System Constraints9 May 2026
- 2026Beyond Factor Aggregation: Gauge-Aware Low-Rank Server Representations for Federated LoRA7 May 2026
- 2026Adaptive Selection of LoRA Components in Privacy-Preserving Federated Learning7 May 2026
- 2026Low Rank Adaptation for Adversarial Perturbation30 Apr 2026
- 2026FedSpy-LLM: Towards Scalable and Generalizable Data Reconstruction Attacks from Gradients on LLMs7 Apr 2026
- 2026FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuning9 Mar 2026
- 2026Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models23 Feb 2026
- 2026Communication-Efficient Personalized Adaptation via Federated-Local Model Merging20 Feb 2026
- 2026FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignment19 Feb 2026
- 2026Preventing Rank Collapse in Federated Low-Rank Adaptation with Client Heterogeneity13 Feb 2026
- 2026Collaborative and Efficient Fine-tuning: Leveraging Task Similarity6 Feb 2026
- 2026TinyGuard:A lightweight Byzantine Defense for Resource-Constrained Federated Learning via Statistical Update Fingerprints2 Feb 2026
- 2026Toward Federated Large Language Models in Medicine: A Parameter-Efficient Framework for Privacy-Preserving, Multi-Institutional Adaptation29 Jan 2026
- 2026SDFLoRA: Selective Decoupled Federated LoRA for Privacy-preserving Fine-tuning with Heterogeneous Clients16 Jan 2026
- 2025FedMPDD: Communication-Efficient Federated Learning with Privacy Preservation Attributes via Projected Directional Derivative23 Dec 2025
- 2025MobileFineTuner: A Mobile-Native Framework for On-Device LLM Fine-Tuning in Real-World Embedded AI Applications9 Dec 2025
- 2025ILoRA: Federated Learning with Low-Rank Adaptation for Heterogeneous Client Aggregation20 Nov 2025
- 2025HFL-FlowLLM: Large Language Models for Network Traffic Flow Classification in Heterogeneous Federated Learning18 Nov 2025
- 2025FedHFT: Efficient Federated Finetuning with Heterogeneous Edge Clients15 Oct 2025
- 2025FedLoDrop: Federated LoRA with Dropout for Generalized LLM Fine-tuning14 Oct 2025
- 2025Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Survey30 Sep 2025
- 2025Communication-Efficient and Accurate Approach for Aggregation in Federated Low-Rank Adaptation30 Sep 2025
- 2025Bi-level Personalization for Federated Foundation Models: A Task-vector Aggregation Approach16 Sep 2025
- 2025A Federated Fine-Tuning Paradigm of Foundation Models in Heterogenous Wireless Networks5 Sep 2025
- 2025Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insights5 Sep 2025
- 2025FediLoRA: Practical Federated Fine-Tuning of Foundation Models Under Missing-Modality Constraints1 Sep 2025
- 2025On the Evolution of Federated Post-Training Large Language Models: A Model Accessibility View22 Aug 2025
50 newest of 114 All 114 (JSON)
- 2026Rethinking LoRA for Privacy-Preserving Federated Learning in Large Modelscited by 4
- 2025Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Surveycited by 2
- 2025Advancing Compositional LLM Reasoning with Structured Task Relations in Interactive Multimodal Communicationscited by 0
- 2026Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approachcited by 0
- 2026Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approach10 Aug 2026
- 2026Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models23 Feb 2026
- 2025Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Survey30 Sep 2025
- 2025Advancing Compositional LLM Reasoning with Structured Task Relations in Interactive Multimodal Communications28 Jul 2025
- 2025FedALT: Federated Fine-Tuning through Adaptive Local Training with Rest-of-World LoRAcited by 13
- 2025FedMerge: Federated Personalization via Model Mergingcited by 4
- 2025FLAME: Towards Federated Fine-Tuning Large Language Models Through Adaptive SMoEcited by 0
- 2026UB-SMoE: Universally Balanced Sparse Mixture-of-Experts for Resource-adaptive Federated Fine-tuning of Foundation Modelscited by 0
- 2026UB-SMoE: Universally Balanced Sparse Mixture-of-Experts for Resource-adaptive Federated Fine-tuning of Foundation Models15 May 2026
- 2025FLAME: Towards Federated Fine-Tuning Large Language Models Through Adaptive SMoE19 Jun 2025
- 2025FedMerge: Federated Personalization via Model Merging9 Apr 2025
- 2025FedALT: Federated Fine-Tuning through Adaptive Local Training with Rest-of-World LoRA14 Mar 2025
- 2026Federated Agent Optimizationcited by 0
- 2026Federated Agent Optimization1 Oct 2026
- 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 144
- 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
- 2024A Survey on LoRA of Large Language Modelscited by 67
- 2024LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinementcited by 37
- 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
- 2024Advances and Open Challenges in Federated Foundation Modelscited by 33
- 2025HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 30
- 2024FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Modelscited by 26
- 2024Federated LoRA with Sparse Communicationcited by 26
- 2024Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clientscited by 22
- 2024An overview of domain-specific foundation model: key technologies, applications and challengescited by 19
- 2024FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Foundation Modelscited by 18
12 most cited of 103 All 103 (JSON)
- 2026Beyond Uniform Compression: Budgeted Transmission Allocation for Extreme Federated Learning30 Sep 2026
- 2026CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation27 Sep 2026
- 2026Perplexity Predicts Protection: Choosing Pretrained Backbones for Worst-Client Fairness in Federated Parameter-Efficient Fine-Tuning20 Sep 2026
- 2026CIDERS: Cloud-Edge LLM Collaborative Learning via Accelerating Personalized Bilevel Optimization14 Sep 2026
- 2026Breaking the Structural Identity: Personalized Federated LoRA Fine-tuning under Rank Heterogeneity1 Sep 2026
- 2026Unveiling the Depth-Performance Dilemma in Split-Federated Fine-tuning of LLMs23 Aug 2026
- 2026FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA15 Aug 2026
- 2026SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Ranks10 Aug 2026
- 2026MineGrad: Gradient Inversion Attacks on LoRA Fine-Tuning2 Aug 2026
- 2026Federated Foundation Models Fine-Tuning with Heterogeneous Compressed Clients31 Jul 2026
- 2026Spectral Transformation for Layer-wise Global Rank Discovery in Federated LoRA for Vision Transformers23 Jul 2026
- 2026Three-Pronged Spectral Control for Federated Parameter Efficient Fine Tuning23 Jul 2026
12 newest of 103 All 103 (JSON)