mlsanityData to 14 Oct 2026

FedSDR: Federated Self-Distillation with Rectification

arXiv 2605.18028 ↗First posted 18 May 2026cs.LG cs.AIcited by 0

On the map: Knowledge distillation → Federated, robust and private distillation

Abstract (arXiv)

Federated fine-tuning of Large Language Models faces severe statistical heterogeneity. However, existing model-level defenses often overlook the root cause: intrinsic data distribution mismatches. In this work, we first establish Federated Self-Distillation (FedSD) as a fundamental and potent strategy. By projecting client representations into a smoothed ``model-understanding space,'' FedSD alone serves as a universal booster, demonstrating superior performance over conventional algorithms. Despite its success, we identify a subtle trade-off termed the Rewrite Paradox -- unconstrained self-distillation can inadvertently increase hallucinations and redundancy. To refine this paradigm, we further propose FedSDR (Federated Self-Distillation with Rectification), the ultimate reinforced framework. It augments FedSD with a dual-stream mechanism: a local LoRA-S (Smoothing) branch to implicitly absorb heterogeneity via distilled data, and a parallel global LoRA-R (Rectification) branch anchored to raw data to enforce factual correctness. By selectively aggregating only LoRA-R, FedSDR yields a globally aligned and faithful model. Extensive experiments verify its superior performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation. Builds on 20 of its 57 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

20 of 57 references.

References · 20

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  4. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  5. 2018Federated Learning with Non-IID Datacited by 1,108
  6. 2020Adaptive Federated Optimizationcited by 1,046
  7. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  8. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  9. 2019Federated Learning with Personalization Layerscited by 541
  10. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 286
  11. 2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuningcited by 96
  12. 2023FinGPT: Democratizing Internet-scale Data for Financial Large Language Modelscited by 67
8 more
  1. 2023pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuningcited by 40
  2. 2024Advances and Open Challenges in Federated Foundation Modelscited by 36
  3. 2024Federated Large Language Models: Current Progress and Future Directionscited by 27
  4. 2024Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architecturescited by 12
  5. 2025Merge-of-Thought Distillationcited by 7
  6. 2024FedDistill: Global Model Distillation for Local Model De-Biasing in Non-IID Federated Learningcited by 4
  7. 2026Training-Trajectory-Aware Token Selectioncited by 4
  8. 2025Co-LoRA: Collaborative Model Personalization on Heterogeneous Multi-Modal Clientscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025FedSODA: Federated Fine-tuning of LLMs via Similarity Group Pruning and Orchestrated Distillation Alignment
  2. 2026FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA
  3. 2025FedALT: Federated Fine-Tuning through Adaptive Local Training with Rest-of-World LoRA
  4. 2024Self-Data Distillation for Recovering Quality in Pruned Large Language Models
  5. 2026FLoKD: Adaptive Knowledge Distillation for Federated Low-Rank LLM over Wireless Networks
  6. 2024Thinking Forward: Memory-Efficient Federated Finetuning of Language Models
  7. 2026CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation
  8. 2025Communication-Aware Knowledge Distillation for Federated LLM Fine-Tuning over Wireless Networks