mlsanityData to 14 Oct 2026

FedSpaLLM: Federated Pruning of Large Language Models

arXiv 2410.14852 ↗First posted 18 Oct 2024cs.LG cs.CRcited by 3

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) achieve state-of-the-art performance but are challenging to deploy due to their high computational and storage demands. Pruning can reduce model size, yet existing methods assume public access to calibration data, which is impractical for privacy-sensitive applications. To address the challenge of pruning LLMs in privacy-preserving settings, we propose FedSpaLLM, the first federated learning framework designed specifically for pruning LLMs. FedSpaLLM enables clients to prune their models locally based on private data while accounting for system heterogeneity and maintaining communication efficiency. Our framework introduces several key innovations: (1) a novel ℓ0-norm aggregation function that ensures only non-zero weights are averaged across clients, preserving important model parameters; (2) an adaptive mask expansion technique that meets global sparsity targets while accommodating client-specific pruning decisions; and (3) a layer sampling strategy that reduces communication overhead and personalizes the pruning process based on client resources. Extensive experiments show that FedSpaLLM improves pruning performance in diverse federated settings.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 27 of its 40 references that are in the map. First posted on arXiv 18 Oct 2024.

Builds on

27 of 40 references.

References · 27

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  9. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  10. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  11. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
  12. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
15 more
  1. 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 222
  2. 2022A Fast Post-Training Pruning Framework for Transformerscited by 162
  3. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 104
  4. 2021Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Maskscited by 103
  5. 2022FedPrompt: Communication-Efficient and Privacy Preserving Prompt Tuning in Federated Learningcited by 91
  6. 2024FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Modelcited by 74
  7. 2024PrE-Text: Training Language Models on Private Federated Data in the Age of LLMscited by 29
  8. 2024FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decompositioncited by 26
  9. 2022FedAdapter: Efficient Federated Learning for Modern NLPcited by 25
  10. 2023AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference with Transformerscited by 24
  11. 2024FedP3: Federated Personalized and Privacy-friendly Network Pruning under Model Heterogeneitycited by 13
  12. 2021FedPrune: Towards Inclusive Federated Learningcited by 12
  13. 2023Complement Sparsification: Low-Overhead Model Pruning for Federated Learningcited by 12
  14. 2022Distributed Pruning Towards Tiny Neural Networks in Federated Learningcited by 11
  15. 2024Secure Federated Learning Across Heterogeneous Cloud and High-Performance Computing Resources -- A Case Study on Federated Fine-tuning of LLaMA 2cited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Exploring Federated Pruning for Large Language Modelscited by 2
  2. 2026Scalable Cross-Facility Federated Learning for Scientific Foundation Models on Multiple Supercomputerscited by 1
  3. 2026CA-AFP: Cluster-Aware Adaptive Federated Pruningcited by 0