mlsanityData to 14 Oct 2026

Emerging Safety Attack and Defense in Federated Instruction Tuning of Large Language Models

arXiv 2406.10630 ↗First posted 15 Jun 2024cs.CL cs.AI cs.CRcited by 18

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Federated learning (FL) enables multiple parties to collaboratively fine-tune an large language model (LLM) without the need of direct data sharing. Ideally, by training on decentralized data that is aligned with human preferences and safety principles, federated instruction tuning can result in an LLM that could behave in a helpful and safe manner. In this paper, we for the first time reveal the vulnerability of safety alignment in FedIT by proposing a simple, stealthy, yet effective safety attack method. Specifically, the malicious clients could automatically generate attack data without involving manual efforts and attack the FedIT system by training their local LLMs on such attack data. Unfortunately, this proposed safety attack not only can compromise the safety alignment of LLM trained via FedIT, but also can not be effectively defended against by many existing FL defense methods. Targeting this, we further propose a post-hoc defense method, which could rely on a fully automated pipeline: generation of defense data and further fine-tuning of the LLM. Extensive experiments show that our safety attack method can significantly compromise the LLM's safety alignment (e.g., reduce safety rate by 70%), which can not be effectively defended by existing defense methods (at most 4% absolute improvement), while our safety defense method can significantly enhance the attacked LLM's safety alignment (at most 69% absolute improvement).

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 56 of its 72 references that are in the map. First posted on arXiv 15 Jun 2024.

Builds on

56 of 72 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  3. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul

Other references · 53

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  10. 2023Mistral 7Bcited by 3,164
  11. 2019Advances and Open Problems in Federated Learningcited by 3,046
  12. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
38 more
  1. 2023BloombergGPT: A Large Language Model for Financecited by 939
  2. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  3. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  4. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  5. 2018Byzantine-Robust Distributed Learning: Towards Optimal Statistical Ratescited by 711
  6. 2023Instruction Tuning with GPT-4cited by 686
  7. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  8. 2019Local Model Poisoning Attacks to Byzantine-Robust Federated Learningcited by 493
  9. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  10. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  11. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  12. 2018Analyzing Federated Learning through an Adversarial Lenscited by 351
  13. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  14. 2019A Little Is Enough: Circumventing Defenses For Distributed Learningcited by 288
  15. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 285
  16. 2020Data Poisoning Attacks Against Federated Learning Systemscited by 275
  17. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 252
  18. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 228
  19. 2018Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learningcited by 225
  20. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 199
  21. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 185
  22. 2020Privacy and Robustness in Federated Learning: Attacks and Defensescited by 166
  23. 2018Mitigating Sybils in Federated Learning Poisoningcited by 161
  24. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 149
  25. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 128
  26. 2020Subpopulation Data Poisoning Attackscited by 70
  27. 2022MPAF: Model Poisoning Attacks to Federated Learning based on Fake Clientscited by 68
  28. 2023FATE-LLM: A Industrial Grade Federated Learning Framework for Large Language Modelscited by 63
  29. 2020Data Poisoning Attacks on Federated Machine Learningcited by 62
  30. 2023No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifiercited by 55
  31. 2023Managing extreme AI risks amid rapid progresscited by 48
  32. 2019Attack-Resistant Federated Learning with Residual-based Reweightingcited by 40
  33. 2023Eliminating Domain Bias for Federated Learning in Representation Spacecited by 38
  34. 2024FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Modelscited by 30
  35. 2024Model Poisoning Attacks to Federated Learning via Multi-Round Consistencycited by 18
  36. 2024Federated Learning with Bilateral Curation for Partially Class-Disjoint Datacited by 13
  37. 2023FedDefender: Client-Side Attack-Tolerant Federated Learningcited by 12
  38. 2024RFLPA: A Robust Federated Learning Framework against Poisoning Attacks with Secure Aggregationcited by 10

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 7; 2026: 6.

18 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  2. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  3. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuningcited by 72
  4. 2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbationcited by 37
  5. 2025Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safetycited by 22
  6. 2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
  7. 2025AsFT: Anchoring Safety During LLM Fine-Tuning Within Narrow Safety Basincited by 19
  8. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  9. 2024PEFT-as-an-Attack! Jailbreaking Language Models during Federated Parameter-Efficient Fine-Tuningcited by 5
  10. 2025Exploring Potential Prompt Injection Attacks in Federated Military LLMs and Their Mitigationcited by 5
  11. 2026Rethinking LoRA for Privacy-Preserving Federated Learning in Large Modelscited by 5
  12. 2024Leveraging Unstructured Text Data for Federated Instruction Tuning of Large Language Modelscited by 2
6 more
  1. 2025Poison Once, Refuse Forever: Weaponizing Alignment for Injecting Bias in LLMscited by 1
  2. 2026Graph Representation Learning Augmented Model Manipulation on Federated Fine-Tuning of LLMscited by 1
  3. 2026Palette: A Modular, Controllable, and Efficient Framework for On-demand Authorized Safety Alignment Relaxation in LLMscited by 1
  4. 2026Learning to Stay Safe: Adaptive Regularization Against Safety Degradation during Fine-Tuningcited by 0
  5. 2026FedDetox: Robust Federated SLM Alignment via On-Device Data Sanitizationcited by 0
  6. 2026Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Safety-Aligned Weights Are Not Enough: Refusal-Teacher-Guided Finetuning Enhances Safety and Downstream Performance under Harmful Finetuning Attacks
  2. 2025Can Federated Learning Safeguard Private Data in LLM Training? Vulnerabilities, Attacks, and Defense Evaluation
  3. 2025Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency
  4. 2025LLM-Safety Evaluations Lack Robustness
  5. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks
  6. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Security
  7. 2024Recent Advances in Federated Learning Driven Large Language Models: A Survey on Architecture, Performance, and Security
  8. 2025SDD: Self-Degraded Defense against Malicious Fine-tuning