mlsanityData to 7 Oct 2026

ParaBlock: Communication-Computation Parallel Block Coordinate Federated Learning for Large Language Models

arXiv 2511.19959 ↗First posted 25 Nov 2025cs.LG cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) has been extensively studied as a privacy-preserving training paradigm. Recently, federated block coordinate descent scheme has become a popular option in training large-scale models, as it allows clients to train only a subset of the model locally instead of the entire model. However, in the era of large language models (LLMs), even a single block can contain a significant number of parameters, posing substantial communication latency, particularly for resource-constrained clients. To address this challenge in federated training/fine-tuning LLMs, we propose ParaBlock, a novel approach that establishes two parallel threads for communication and computation to enhance communication efficiency. We theoretically prove that the proposed ParaBlock achieves the same convergence rate as the standard federated block coordinate descent methods. Empirical evaluations on fine-tuning LLMs on general instruction following and mathematical reasoning confirm that ParaBlock not only maintains strong performance but also significantly improves communication efficiency.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 37 of its 55 references that are in the map. First posted on arXiv 25 Nov 2025.

Builds on

37 of 55 references.

References · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  7. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  9. 2020Adaptive Federated Optimizationcited by 1,036
  10. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  11. 2023Instruction Tuning with GPT-4cited by 676
  12. 2020Federated Learning with Matched Averagingcited by 564
25 more
  1. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  2. 2020Flower: A Friendly Federated Learning Research Frameworkcited by 449
  3. 2019FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantizationcited by 349
  4. 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 219
  5. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 192
  6. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 188
  7. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 185
  8. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 184
  9. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 158
  10. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 144
  11. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 124
  12. 2019Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronizationcited by 106
  13. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 92
  14. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 88
  15. 2021FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Taskscited by 84
  16. 2021Adaptive Quantization of Model Updates for Communication-Efficient Federated Learningcited by 60
  17. 2022Communication-Efficient Adaptive Federated Learningcited by 55
  18. 2022Sparse Random Networks for Communication-Efficient Federated Learningcited by 32
  19. 2024FedMKT: Federated Mutual Knowledge Transfer for Large and Small Language Modelscited by 19
  20. 2020Federated Block Coordinate Descent Scheme for Learning Global and Personalized Modelscited by 11
  21. 2024FADAS: Towards Federated Adaptive Asynchronous Optimizationcited by 10
  22. 2024Privacy Preserving Federated Learning in Medical Imaging with Uncertainty Estimationcited by 9
  23. 2024FedBAT: Communication-Efficient Federated Learning via Learnable Binarizationcited by 9
  24. 2024CG-FedLLM: How to Compress Gradients in Federated Fune-tuning for Large Language Modelscited by 4
  25. 2024Save It All: Enabling Full Parameter Tuning for Federated Large Language Models via Cycle Block Gradient Descentcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Thinking Forward: Memory-Efficient Federated Finetuning of Language Models
  2. 2025FedSEA-LLaMA: A Secure, Efficient and Adaptive Federated Splitting Framework for Large Language Models
  3. 2025FedNano: Toward Lightweight Federated Tuning for Pretrained Multimodal Large Language Models
  4. 2026A Step Toward Federated Pretraining of Multimodal Large Language Models
  5. 2026Conflict-Aware Federated Fine-Tuning of Large Language Models with Mixture-of-Experts
  6. 2026Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions
  7. 2026Pretraining A Large Language Model using Distributed GPUs: A Memory-Efficient Decentralized Paradigm
  8. 2026AlignFed: Alignment-Aware Asynchronous Federated Fine-Tuning for Large Language Models in Heterogeneous Edge Environments