mlsanityData to 13 Oct 2026

C2A: Client-Customized Adaptation for Parameter-Efficient Federated Learning

arXiv 2411.00311 ↗First posted 1 Nov 2024cs.LG cs.AI cs.CRcited by 8

Not on the map. Why?

Abstract (arXiv)

Despite the versatility of pre-trained language models (PLMs) across domains, their large memory footprints pose significant challenges in federated learning (FL), where the training model has to be distributed between a server and clients. One potential solution to bypass such constraints might be the use of parameter-efficient fine-tuning (PEFT) in the context of FL. However, we have observed that typical PEFT tends to severely suffer from heterogeneity among clients in FL scenarios, resulting in unstable and slow convergence. In this paper, we propose Client-Customized Adaptation (C2A), a novel hypernetwork-based FL framework that generates client-specific adapters by conditioning the client information. With the effectiveness of the hypernetworks in generating customized weights through learning to adopt the different characteristics of inputs, C2A can maximize the utility of shared model parameters while minimizing the divergence caused by client heterogeneity. To verify the efficacy of C2A, we perform extensive evaluations on FL scenarios involving heterogeneity in label and language distributions. Comprehensive evaluation results clearly support the superiority of C2A in terms of both efficiency and effectiveness in FL scenarios.

Key facts. Cited by 8 papers in the map (data through 13 Oct 2026). Builds on 29 of its 39 references that are in the map. First posted on arXiv 1 Nov 2024.

Builds on

29 of 39 references.

References · 29

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,653
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,281
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,213
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,916
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,898
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,785
  7. 2020Extracting Training Data from Large Language Modelscited by 2,181
  8. 2019Similarity of Neural Network Representations Revisitedcited by 1,719
  9. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,470
  10. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,248
  11. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,240
  12. 2021Model-Contrastive Federated Learningcited by 766
17 more
  1. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 459
  2. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 320
  3. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 281
  4. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  5. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 222
  6. 2019Identifying and Reducing Gender Bias in Word-Level Language Modelscited by 214
  7. 2021Improving Calibration for Long-Tailed Recognitioncited by 212
  8. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  9. 2019What Would Elsa Do? Freezing Layers During Transformer Fine-Tuningcited by 88
  10. 2022HyperPrompt: Prompt-based Task-Conditioning of Transformerscited by 67
  11. 2019How Does BERT Answer Questions? A Layer-Wise Analysis of Transformer Representationscited by 63
  12. 2020What's so special about BERT's layers? A closer look at the NLP pipeline in monolingual and multilingual modelscited by 35
  13. 2021Exploiting Adapters for Cross-lingual Low-resource Speech Recognitioncited by 34
  14. 2019Federated Learning of N-gram Language Modelscited by 33
  15. 2022Pretrained Models for Multilingual Federated Learningcited by 30
  16. 2021FedMatch: Federated Learning Over Heterogeneous Question Answering Datacited by 23
  17. 2022Hyper-X: A Unified Hypernetwork for Multi-Task Multilingual Transfercited by 22

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 7; 2026: 1.

8 papers.

  1. 2025A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learningcited by 17
  2. 2025Probabilistic Federated Prompt-Tuning with Non-IID and Imbalanced Datacited by 9
  3. 2025FLoRIST: Singular Value Thresholding for Efficient and Accurate Federated Fine-Tuning of Large Language Modelscited by 4
  4. 2025Learning Like Humans: Resource-Efficient Federated Fine-Tuning through Cognitive Developmental Stagescited by 4
  5. 2025Gradient Inversion Attacks on Parameter-Efficient Fine-Tuningcited by 3
  6. 2025Assortment of Attention Heads: Accelerating Federated PEFT with Head Pruning and Strategic Client Selectioncited by 0
  7. 2025FLAME: Towards Federated Fine-Tuning Large Language Models Through Adaptive SMoEcited by 0
  8. 2026MineGrad: Gradient Inversion Attacks on LoRA Fine-Tuningcited by 0