mlsanityData to 14 Oct 2026

FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learning

arXiv 2108.06098 ↗First posted 13 Aug 2021cs.LG cs.CVcited by 141

Not on the map. Why?

Abstract (arXiv)

In this work, we propose a communication-efficient parameterization, FedPara, for federated learning (FL) to overcome the burdens on frequent model uploads and downloads. Our method re-parameterizes weight parameters of layers using low-rank weights followed by the Hadamard product. Compared to the conventional low-rank parameterization, our FedPara method is not restricted to low-rank constraints, and thereby it has a far larger capacity. This property enables to achieve comparable performance while requiring 3 to 10 times lower communication costs than the model with the original layers, which is not achievable by the traditional low-rank methods. The efficiency of our method can be further improved by combining with other efficient FL optimizers. In addition, we extend our method to a personalized FL application, pFedPara, which separates parameters into global and local ones. We show that pFedPara outperforms competing personalized FL methods with more than three times fewer parameters.

Key facts. Cited by 141 papers in the map (data through 14 Oct 2026). Builds on 35 of its 51 references that are in the map. First posted on arXiv 13 Aug 2021.

Builds on

35 of 51 references.

References · 35

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  6. 2019Advances and Open Problems in Federated Learningcited by 3,046
  7. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  8. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,238
  9. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  10. 2020Adaptive Federated Optimizationcited by 1,046
  11. 2018LEAF: A Benchmark for Federated Settingscited by 792
  12. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
23 more
  1. 2019Federated Learning with Personalization Layerscited by 541
  2. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 478
  3. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 442
  4. 2019FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantizationcited by 356
  5. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
  6. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 305
  7. 2020FedML: A Research Library and Benchmark for Federated Machine Learningcited by 300
  8. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 256
  9. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 238
  10. 2018The Convergence of Sparsified Gradient Methodscited by 229
  11. 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 203
  12. 2020Federated Learning with Compression: Unified Analysis and Sharp Guaranteescited by 160
  13. 2016QSGD: Communication-Efficient SGD via Gradient Quantization and Encodingcited by 147
  14. 2020A Survey of Privacy Attacks in Machine Learningcited by 146
  15. 2020One-Bit Over-the-Air Aggregation for Communication-Efficient Federated Edge Learning: Design and Convergence Analysiscited by 112
  16. 2015Convolutional neural networks with low-rank regularizationcited by 103
  17. 2020Federated Accelerated Stochastic Gradient Descentcited by 85
  18. 2020Stable Low-rank Tensor Decomposition for Compression of Convolutional Neural Networkcited by 74
  19. 2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
  20. 2020Federated Learning with Only Positive Labelscited by 55
  21. 2017Introduction to Tensor Decompositions and their Applications in Machine Learningcited by 46
  22. 2020Accordion: Adaptive Gradient Communication via Critical Learning Regime Identificationcited by 19
  23. 2021Communication-Efficient Federated Learning with Dual-Side Low-Rank Compressioncited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 5; 2023: 13; 2024: 59; 2025: 42; 2026: 20.

141 papers.

Sort:

By area:

  1. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  2. 2023One-Dimensional Adapter to Rule Them All: Concepts, Diffusion Models and Erasing Applicationscited by 169
  3. 2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 161
  4. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 147
  5. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 141
  6. 2023Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluationcited by 109
  7. 2024LoRA-Pro: Are Low-Rank Adapters Properly Optimized?cited by 77
  8. 2024SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)cited by 59
  9. 2023UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformerscited by 54
  10. 2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptationcited by 48
  11. 2023CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformerscited by 44
  12. 2023Personalized Federated Learning under Mixture of Distributionscited by 37
38 more
  1. 2021Layer-wise Adaptive Model Aggregation for Scalable Federated Learningcited by 36
  2. 2024Relaxed Contrastive Learning for Federated Learningcited by 32
  3. 2024Customizing Language Models with Instance-wise LoRA for Sequential Recommendationcited by 32
  4. 2021FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorizationcited by 31
  5. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  6. 2025HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 31
  7. 2024Harmonizing Generalization and Personalization in Federated Prompt Learningcited by 27
  8. 2023Cuttlefish: Low-Rank Model Training without All the Tuningcited by 25
  9. 2022Accelerated Federated Learning with Decoupled Adaptive Optimizationcited by 23
  10. 2024QuanTA: Efficient High-Rank Fine-Tuning of LLMs with Quantum-Informed Tensor Adaptationcited by 23
  11. 2024ResLoRA: Identity Residual Mapping in Low-Rank Adaptioncited by 18
  12. 2024Recovering the Pre-Fine-Tuning Weights of Generative Modelscited by 16
  13. 2023Reusing Pretrained Models by Multi-linear Operators for Efficient Trainingcited by 15
  14. 2024Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptationcited by 15
  15. 2024Towards Efficient Communication and Secure Federated Recommendation System via Low-rank Trainingcited by 14
  16. 2024See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decompositioncited by 14
  17. 2024Propulsion: Steering LLM with Tiny Fine-Tuningcited by 14
  18. 2022Federated Learning of Large Models at the Edge via Principal Sub-Model Trainingcited by 13
  19. 2024Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferencescited by 13
  20. 2025Ultra-Resolution Adaptation with Easecited by 13
  21. 2024LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Modelscited by 12
  22. 2024LoBAM: LoRA-Based Backdoor Attack on Model Mergingcited by 12
  23. 2025Quantum-PEFT: Ultra parameter-efficient fine-tuningcited by 11
  24. 2024Parameter-Efficient Fine-Tuning of Large Language Models using Semantic Knowledge Tuningcited by 10
  25. 2025Hierarchical LoRA MoE for Efficient CTR Model Scalingcited by 10
  26. 2024Bayesian Parameter-Efficient Fine-Tuning for Overcoming Catastrophic Forgettingcited by 9
  27. 2024DP-DyLoRA: Fine-Tuning Transformer-Based Models On-Device under Differentially Private Federated Learning using Dynamic Low-Rank Adaptationcited by 9
  28. 2024FedBAT: Communication-Efficient Federated Learning via Learnable Binarizationcited by 9
  29. 2024TPP-LLM: Modeling Temporal Point Processes by Efficiently Fine-Tuning Large Language Modelscited by 9
  30. 2025Hadamard product in deep learning: Introduction, Advances and Challengescited by 9
  31. 2025AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projectionscited by 9
  32. 2025RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Modelscited by 9
  33. 2025Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuningcited by 9
  34. 2023Making Batch Normalization Great in Federated Deep Learningcited by 8
  35. 2024RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updatescited by 8
  36. 2024Federated Dynamical Low-Rank Training with Global Loss Convergence Guaranteescited by 7
  37. 2024Dataset Size Recovery from LoRA Weightscited by 7
  38. 2024BA-LoRA: Bias-Alleviating Low-Rank Adaptation to Mitigate Catastrophic Inheritance in Large Language Modelscited by 7

50 most cited of 141 All 141 (JSON)