mlsanityData to 7 Oct 2026

FLoCoRA: Federated learning compression with low-rank adaptation

arXiv 2406.14082 ↗First posted 20 Jun 2024cs.LG eess.SPcited by 3

Not on the map. Why?

Abstract (arXiv)

Low-Rank Adaptation (LoRA) methods have gained popularity in efficient parameter fine-tuning of models containing hundreds of billions of parameters. In this work, instead, we demonstrate the application of LoRA methods to train small-vision models in Federated Learning (FL) from scratch. We first propose an aggregation-agnostic method to integrate LoRA within FL, named FLoCoRA, showing that the method is capable of reducing communication costs by 4.8 times, while having less than 1% accuracy degradation, for a CIFAR-10 classification task with a ResNet-8. Next, we show that the same method can be extended with an affine quantization scheme, dividing the communication cost by 18.6 times, while comparing it with the standard method, with still less than 1% of accuracy loss, tested with on a ResNet-18 model. Our formulation represents a strong baseline for message size reduction, even when compared to conventional model compression works, while also reducing the training memory requirements due to the low-rank adaptation.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 23 of its 24 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

23 of 24 references.

References · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2019Advances and Open Problems in Federated Learningcited by 3,021
  5. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  6. 2019Learned Step Size Quantizationcited by 604
  7. 2021Towards Personalized Federated Learningcited by 462
  8. 2021A White Paper on Neural Network Quantizationcited by 453
  9. 2020What is being transferred in transfer learning?cited by 365
  10. 2019FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantizationcited by 349
  11. 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 298
  12. 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 219
11 more
  1. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 150
  2. 2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 140
  3. 2022Overcoming Oscillations in Quantization-Aware Trainingcited by 116
  4. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 114
  5. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 92
  6. 2018Training and Inference with Integers in Deep Neural Networkscited by 84
  7. 2022ZeroFL: Efficient On-Device Training for Federated Learning with Local Sparsitycited by 45
  8. 2023Unified Data-Free Compression: Pruning and Quantization without Fine-Tuningcited by 22
  9. 2024A Novel Benchmark for Few-Shot Semantic Segmentation in the Era of Foundation Modelscited by 13
  10. 2020Rethinking Weight Decay For Efficient Neural Network Pruningcited by 10
  11. 2023Federated learning compression designed for lightweight communicationscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Frugal Federated Learning for Violence Detection: A Comparison of LoRA-Tuned VLMs and Personalized CNNscited by 0
  2. 2025Federated Learning for Video Violence Detection: Complementary Roles of Lightweight CNNs and Vision-Language Models for Energy-Efficient Usecited by 0
  3. 2026FedSpy-LLM: Towards Scalable and Generalizable Data Reconstruction Attacks from Gradients on LLMscited by 0