mlsanityData to 14 Oct 2026

Priority-Aware Learning-Unlearning Correction for Dynamic Decentralized LoRA Fine-Tuning

arXiv 2606.22878 ↗First posted 22 Jun 2026cs.LG cs.AIcited by 0

On the map: LLM safety and security → Federated unlearning and poisoning

Abstract (arXiv)

As large language models (LLMs) are increasingly deployed at the network edge to provide pervasive generative AI services, decentralized federated learning (DFL) provides a vital mechanism for privacy-preserving, domain-specific fine-tuning through peer-to-peer exchanges of parameter-efficient updates. However, the dynamic nature of practical decentralized edge networks, where devices may dynamically join or leave the collaborative training process, requires the system to continuously adapt to new data while selectively removing prior contributions. This correction process remains a significant bottleneck, as individual device updates become deeply entangled within the global fine-tuned parameters. To address this challenge, we propose a priority-aware learning-unlearning correction framework based on orthogonal LoRA that can enhance the knowledge evaluation through topology adjustment. Specifically, we first design an orthogonal LoRA mechanism that yields post-training contribution coordinates, enabling history-free projection addition and deletion in response to membership changes. We then analyze the correction bottleneck and develop a priority-aware policy that selects among topology refinement, local correction, proximal damping, and synchronization scheduling according to the dominant residual term. A resource allocation algorithm is further developed to allocate limited communication across layer groups, prioritizing the primary bottlenecks within per-round wireless constraints. Experiments demonstrate that the proposed framework achieves robust post-event correction for both device join and leave events and validate that different residual regimes necessitate distinct correction actions.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Federated unlearning and poisoning, one of 12 directions in LLM safety and security. Builds on 71 of its 102 references that are in the map. First posted on arXiv 22 Jun 2026.

Builds on

71 of 102 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026FedCARE: Federated Unlearning with Conflict-Aware Projection and Relearning-Resistant Recovery30 Jan
  2. 2026Certified Unlearning in Decentralized Federated Learning10 Jan
  3. 2024Federated Unlearning with Gradient Descent and Conflict Mitigation28 Dec
  4. 2024Enable the Right to be Forgotten with Federated Client Unlearning in Medical Imaging2 Jul
  5. 2024Fast-FedUL: A Training-Free Federated Unlearning with Provable Skew Resilience28 May
  6. 2023A Survey on Federated Unlearning: Challenges, Methods, and Future Directions31 Oct
  7. 2022Federated Unlearning with Knowledge Distillation24 Jan
  8. 2021Bayesian Variational Federated Learning and Unlearning in Decentralized Networks8 Apr

Other references · 63

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  7. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  8. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  9. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
  10. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 502
  11. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
  12. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
38 more
  1. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 312
  2. 2019A Joint Learning and Communications Framework for Federated Learning over Wireless Networkscited by 263
  3. 2024Rethinking Machine Unlearning for Large Language Modelscited by 257
  4. 2013On the Convergence of Decentralized Gradient Descentcited by 159
  5. 2021Distributed Learning in Wireless Networks: Recent Progress and Future Challengescited by 129
  6. 2017Network Topology and Communication-Computation Tradeoffs in Decentralized Optimizationcited by 122
  7. 2019Optimal Wireless Resource Allocation with Random Edge Graph Neural Networkscited by 90
  8. 2020Graph Neural Networks for Scalable Radio Resource Management: Architecture Design and Theoretical Analysiscited by 81
  9. 2022FedRecover: Recovering from Poisoning Attacks in Federated Learning using Historical Informationcited by 58
  10. 2021Decentralized Federated Learning through Proxy Model Sharingcited by 57
  11. 2021Decentralized Federated Learning: Balancing Communication and Computing Costscited by 55
  12. 2024On Large Language Model Continual Unlearningcited by 52
  13. 2020Wireless Communications for Collaborative Federated Learningcited by 47
  14. 2022Graph Neural Networks for Wireless Communications: From Theory to Practicecited by 46
  15. 2021Opportunities of Federated Learning in Connected, Cooperative and Automated Industrial Systemscited by 45
  16. 2024LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinementcited by 38
  17. 2023Learn to Unlearn for Deep Neural Networks: Minimizing Unlearning Interference with Gradient Projectioncited by 37
  18. 2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 35
  19. 2019Graph Embedding based Wireless Link Scheduling with Few Training Samplescited by 30
  20. 2020Byzantine-resilient Decentralized Stochastic Gradient Descentcited by 28
  21. 2023Large Language Models for Networking: Applications, Enabling Techniques, and Challengescited by 28
  22. 2022Beyond spectral gap: The role of the topology in decentralized learningcited by 27
  23. 2025LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptationcited by 19
  24. 2021Decentralized Wireless Federated Learning with Differential Privacycited by 18
  25. 2021Decentralized and Model-Free Federated Learning: Consensus-Based Distillation in Function Spacecited by 17
  26. 2025OMoE: Diversifying Mixture of Low-Rank Adaptation by Orthogonal Finetuningcited by 15
  27. 2025Decentralized Low-Rank Fine-Tuning of Large Language Modelscited by 14
  28. 2025FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRAcited by 11
  29. 2025FedQuad: Adaptive Layer-wise LoRA Deployment and Activation Quantization for Federated Fine-Tuningcited by 10
  30. 2026Token-Level LLM Collaboration via FusionRoutecited by 9
  31. 2023Towards Cooperative Federated Learning over Heterogeneous Edge/Fog Networkscited by 8
  32. 2025Federated Fine-Tuning of LLMs: Framework Comparison and Research Directionscited by 8
  33. 2016Expander Graph and Communication-Efficient Decentralized Optimizationcited by 7
  34. 2023The Role of Federated Learning in a Wireless World with Foundation Modelscited by 7
  35. 2025Fed-HeLLo: Efficient Federated Foundation Model Fine-Tuning with Heterogeneous LoRA Allocationcited by 7
  36. 2025FFT-MoE: Efficient Federated Fine-Tuning for Foundation Models via Large-scale Sparse MoE under Heterogeneous Edgecited by 7
  37. 2021Network Support for High-performance Distributed Machine Learningcited by 4
  38. 2021Wireless Link Scheduling via Graph Representation Learning: A Comparative Study of Different Supervision Levelscited by 4

50 most cited of 63 All 63 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026FLoKD: Adaptive Knowledge Distillation for Federated Low-Rank LLM over Wireless Networks
  2. 2026Federated LoRA Fine-Tuning for LLMs via Collaborative Alignment
  3. 2025FedALT: Federated Fine-Tuning through Adaptive Local Training with Rest-of-World LoRA
  4. 2026CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation
  5. 2026Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models
  6. 2025Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning
  7. 2026PFAdapter: Hierarchical LoRA Decomposition for Personalized Federated MLLMs
  8. 2025Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning