FLoKD: Adaptive Knowledge Distillation for Federated Low-Rank LLM over Wireless Networks
On the map: Knowledge distillation → Federated, robust and private distillation
Abstract (arXiv)
Large language models (LLMs) have demonstrated strong capabilities across a wide range of natural language processing tasks. However, conventional fine-tuning typically relies on centralized data collection, bringing in privacy concerns. Federated learning (FL) enables collaborative LLM fine-tuning without sharing raw client data, but its deployment over bandwidth-constrained wireless networks is hindered by the communication overhead of model-parameter transmission. Although Low-Rank Adaptation (LoRA) reduces the number of trainable parameters, its communication cost still increases with model scale. Knowledge distillation avoids parameter sharing via output logits, but token-level logits in LLMs incur high communication cost due to sequence length and vocabulary size. Reducing logits lowers the cost but weakens supervision and degrades accuracy. To address these limitations, we propose FLoKD, an adaptive knowledge-distillation framework for federated LoRA fine-tuning of LLMs over wireless networks, which communicates intermediate LoRA activations as the distillation signal rather than logits or full parameters. Since transmitting all blocks over the entire public dataset remains costly, we further propose a transformer block importance scoring framework that selectively transmits the most informative blocks, and two dataset selection strategies that discard public samples deviating from the local data distribution and prioritise those most informative for distillation. Extensive experiments across multiple generative language datasets, including WikiText-103, PTB, and Dialog, demonstrate that our proposed framework reduces communication overhead by 50-65% while achieving rapid convergence to competitive perplexity compared to baselines.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation. Builds on 28 of its 37 references that are in the map. First posted on arXiv 11 Sep 2026.
Builds on
28 of 37 references.
Closest ancestors · 1
Same direction, newest first
- 2025Communication-Aware Knowledge Distillation for Federated LLM Fine-Tuning over Wireless Networks1 Sep
Other references · 27
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2016Pointer Sentinel Mixture Modelscited by 3,096
- 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
- 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
- 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
- 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 652
- 2019Importance Estimation for Neural Network Pruningcited by 541
- 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
- 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
15 more
- 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
- 2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 75
- 2020FedAT: A High-Performance and Communication-Efficient Federated Learning System with Asynchronous Tierscited by 64
- 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 45
- 2024DDK: Distilling Domain Knowledge for Efficient Large Language Modelscited by 26
- 2024FedMKT: Federated Mutual Knowledge Transfer for Large and Small Language Modelscited by 19
- 2025A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learningcited by 17
- 2025A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clonecited by 15
- 2025LayerIF: Estimating Layer Quality for Large Language Models using Influence Functionscited by 12
- 2024TraKDis: A Transformer-based Knowledge Distillation Approach for Visual Reinforcement Learning with Application to Cloth Manipulationcited by 8
- 2024Joint Model Pruning and Resource Allocation for Wireless Time-triggered Federated Learningcited by 3
- 2024HAFLQ: Heterogeneous Adaptive Federated LoRA Fine-tuned LLM with Quantizationcited by 2
- 2023Distributed Learning in Heterogeneous Environment: federated learning with adaptive aggregation and computation reductioncited by 1
- 2025TT-Prune: Joint Model Pruning and Resource Allocation for Communication-efficient Time-triggered Federated Learningcited by 1
- 2026CRB-Guided Sensing and Resource Allocation for Human Pose Prediction in Integrated Sensing, Communication, and Computation Systemscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Wireless Federated Multi-Task LLM Fine-Tuning via Sparse-and-Orthogonal LoRA
- 2026CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation
- 2025FedALT: Federated Fine-Tuning through Adaptive Local Training with Rest-of-World LoRA
- 2025Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning
- 2024Personalized Wireless Federated Learning for Large Language Models
- 2025FedSODA: Federated Fine-tuning of LLMs via Similarity Group Pruning and Orchestrated Distillation Alignment
- 2026Federated LoRA Fine-Tuning for LLMs via Collaborative Alignment
- 2025Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning