mlsanityData to 14 Oct 2026

Assortment of Attention Heads: Accelerating Federated PEFT with Head Pruning and Strategic Client Selection

arXiv 2506.00743 ↗First posted 31 May 2025cs.CL cs.AI cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

Parameter Efficient Fine-Tuning (PEFT) has become the de-facto approach in adapting Large Language Models (LLMs) for downstream tasks in Natural Language Processing. However, its adoption in privacy-preserving distributed learning frameworks, such as Federated Learning (FL), remains relatively limited. This is mainly due to challenges specific to FL, such as resource-constrained devices and diverse data distributions among clients. In this paper, we propose an efficient method to perform PEFT within the FL framework for Multi-Head Attention (MHA) based language models. We address the challenges through head pruning, a novel head-specific weighted aggregation mechanism, and a client selection strategy. Head pruning minimizes training complexity within the clients, guided by the importance score computed based on the confidence of the attention head. Weighted aggregation of heads ensures the global model captures crucial updates from diverse clients complementing our client selection strategy. We show results on the MultiNLI benchmark along with 20 Newsgroups, XL-Sum, and E2E NLG datasets. We use the MultiNLI dataset and T5-small model with LoRA as our PEFT method, attaining sparsity levels of up to 90%, resulting in a communication advantage of up to 1.8x and a reduction in training OPs of 3.9x while maintaining the accuracy drop under 2%.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 54 of its 68 references that are in the map. First posted on arXiv 31 May 2025.

Builds on

54 of 68 references.

References · 54

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  9. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  10. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  11. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  12. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
38 more
  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  5. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  6. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  7. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  8. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  9. 2021GPT Understands, Toocited by 716
  10. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  11. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  12. 2020What is the State of Neural Network Pruning?cited by 583
  13. 2018Federated Learning for Mobile Keyboard Predictioncited by 572
  14. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 553
  15. 2018Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edgecited by 352
  16. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  17. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 253
  18. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  19. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 230
  20. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 185
  21. 2021Block Pruning For Faster Transformerscited by 177
  22. 2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 141
  23. 2021Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learningcited by 111
  24. 2020Client Selection and Bandwidth Allocation in Wireless Federated Learning Networks: A Long-Term Perspectivecited by 103
  25. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 93
  26. 2021FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Taskscited by 85
  27. 2023When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directionscited by 84
  28. 2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 75
  29. 2020An Efficiency-boosting Client Selection Scheme for Federated Learning with Fairness Guaranteecited by 62
  30. 2022Client Selection in Federated Learning: Principles, Challenges, and Opportunitiescited by 62
  31. 2021Differentiable Subset Pruning of Transformer Headscited by 47
  32. 2021Federated Learning Meets Natural Language Processing: A Surveycited by 43
  33. 2021Scaling Federated Learning for Fine-tuning of Large Language Modelscited by 36
  34. 2022FedTP: Federated Learning by Transformer Personalizationcited by 36
  35. 2019Federated Learning of N-gram Language Modelscited by 33
  36. 2022FedTune: A Deep Dive into Efficient Federated Fine-Tuning with Pre-trained Transformerscited by 33
  37. 2024FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decompositioncited by 26
  38. 2023FedPerfix: Towards Partial Model Personalization of Vision Transformers in Federated Learningcited by 21

50 most cited of 54 All 54 (JSON)

Built on by

Not cited yet.