mlsanityData to 7 Oct 2026

FedRA: A Random Allocation Strategy for Federated Tuning to Unleash the Power of Heterogeneous Clients

arXiv 2311.11227 ↗First posted 19 Nov 2023cs.LG cs.AI cs.DCcited by 18

Not on the map. Why?

Abstract (arXiv)

With the increasing availability of Foundation Models, federated tuning has garnered attention in the field of federated learning, utilizing data and computation resources from multiple clients to collaboratively fine-tune foundation models. However, in real-world federated scenarios, there often exist a multitude of heterogeneous clients with varying computation and communication resources, rendering them incapable of supporting the entire model fine-tuning process. In response to this challenge, we propose a novel federated tuning algorithm, FedRA. The implementation of FedRA is straightforward and can be seamlessly integrated into any transformer-based model without the need for further modification to the original model. Specifically, in each communication round, FedRA randomly generates an allocation matrix. For resource-constrained clients, it reorganizes a small number of layers from the original model based on the allocation matrix and fine-tunes using adapters. Subsequently, the server aggregates the updated adapter parameters from the clients according to the current allocation matrix into the corresponding layers of the original model. It is worth noting that FedRA also supports scenarios where none of the clients can support the entire global model, which is an impressive advantage. We conduct experiments on two large-scale image datasets, DomainNet and NICO++, under various non-iid settings. The results demonstrate that FedRA outperforms the compared methods significantly. The source code is available at urlhttps://github.com/leondada/FedRA.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 27 of its 38 references that are in the map. First posted on arXiv 19 Nov 2023.

Builds on

27 of 38 references.

References · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  6. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  7. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  8. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  9. 2019Towards Federated Learning at Scale: System Designcited by 826
  10. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
  11. 2021Federated Learning Based on Dynamic Regularizationcited by 555
  12. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 537
15 more
  1. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 334
  2. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  3. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 133
  4. 2022NICO++: Towards Better Benchmarking for Domain Generalizationcited by 86
  5. 2023FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learningcited by 70
  6. 2023FATE-LLM: A Industrial Grade Federated Learning Framework for Large Language Modelscited by 62
  7. 2023Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generationcited by 54
  8. 2023Learning Federated Visual Prompt in Null Space for MRI Reconstructioncited by 40
  9. 2022No One Left Behind: Inclusive Federated Learning over Heterogeneous Devicescited by 36
  10. 2022Disentangled Federated Learning for Tackling Attributes Skew via Invariant Aggregation and Diversity Transferringcited by 28
  11. 2016Challenges and Opportunities in Edge Computingcited by 26
  12. 2023GPT-FL: Generative Pre-trained Model-Assisted Federated Learningcited by 24
  13. 2023Visual Prompt Based Personalized Federated Learningcited by 16
  14. 2023Efficient Federated Prompt Tuning for Black-box Large Pre-trained Modelscited by 12
  15. 2023Towards Instance-adaptive Inference for Federated Learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 7; 2026: 4.

18 papers.

Sort:

By area:

  1. 2024LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinementcited by 37
  2. 2024Advances and Open Challenges in Federated Foundation Modelscited by 33
  3. 2025A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learningcited by 15
  4. 2024Synergizing Foundation Models and Federated Learning: A Surveycited by 14
  5. 2024Federated Continual Learning for Edge-AI: A Comprehensive Surveycited by 10
  6. 2025Federated Low-Rank Adaptation for Foundation Models: A Surveycited by 9
  7. 2025Fed-HeLLo: Efficient Federated Foundation Model Fine-Tuning with Heterogeneous LoRA Allocationcited by 7
  8. 2024PEFT-as-an-Attack! Jailbreaking Language Models during Federated Parameter-Efficient Fine-Tuningcited by 5
  9. 2025Adaptive Rank Allocation for Federated Parameter-Efficient Fine-Tuning of Language Modelscited by 4
  10. 2024Fed-pilot: Optimizing LoRA Allocation for Efficient Federated Fine-Tuning with Heterogeneous Clientscited by 3
  11. 2026FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuningcited by 3
  12. 2024EFTViT: Efficient Federated Training of Vision Transformers with Masked Images on Resource-Constrained Clientscited by 2
6 more
  1. 2026DSFedMed: Dual-Scale Federated Medical Image Segmentation via Mutual Distillation Between Foundation and Lightweight Modelscited by 1
  2. 2025FedSCA: Federated Tuning with Similarity-guided Collaborative Aggregation for Heterogeneous Medical Image Segmentationcited by 0
  3. 2025Decentralized Rank Scheduling for Energy-Constrained Multi-Task Federated Fine-Tuning in Edge-Assisted IoV Networkscited by 0
  4. 2025ILoRA: Federated Learning with Low-Rank Adaptation for Heterogeneous Client Aggregationcited by 0
  5. 2026Fed-DLoRA: Efficient Wireless Federated Learning with Dynamic Low-Rank Adaptationcited by 0
  6. 2026FedSmoothLoRA: Toward Smoother and Faster Convergence in Federated Low-Rank Adaptationcited by 0