mlsanityData to 14 Oct 2026

Poison Once, Refuse Forever: Weaponizing Alignment for Injecting Bias in LLMs

arXiv 2508.20333 ↗First posted 28 Aug 2025cs.LG cs.AI cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) are aligned to meet ethical standards and safety requirements by training them to refuse answering harmful or unsafe prompts. In this paper, we demonstrate how adversaries can exploit LLMs' alignment to implant bias, or enforce targeted censorship without degrading the model's responsiveness to unrelated topics. Specifically, we propose Subversive Alignment Injection (SAI), a poisoning attack that leverages the alignment mechanism to trigger refusal on specific topics or queries predefined by the adversary. Although it is perhaps not surprising that refusal can be induced through overalignment, we demonstrate how this refusal can be exploited to inject bias into the model. Surprisingly, SAI evades state-of-the-art poisoning defenses including LLM state forensics, as well as robust aggregation techniques that are designed to detect poisoning in FL settings. We demonstrate the practical dangers of this attack by illustrating its end-to-end impacts on LLM-powered application pipelines. For chat based applications such as ChatDoctor, with 1% data poisoning, the system refuses to answer healthcare questions to targeted racial category leading to high bias (Delta DP of 23%). We also show that bias can be induced in other NLP tasks: for a resume selection pipeline aligned to refuse to summarize CVs from a selected university, high bias in selection (Delta DP of 27%) results. Even higher bias (Delta DP~38%) results on 9 other chat based downstream applications.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 58 of its 82 references that are in the map. First posted on arXiv 28 Aug 2025.

Builds on

58 of 82 references.

References · 58

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  7. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  8. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  9. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,106
  10. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  11. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  12. 2018Byzantine-Robust Distributed Learning: Towards Optimal Statistical Ratescited by 711
38 more
  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  2. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  3. 2023Bias and Fairness in Large Language Models: A Surveycited by 639
  4. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
  5. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
  6. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  7. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  8. 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 325
  9. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 290
  10. 2023Poisoning Language Models During Instruction Tuningcited by 269
  11. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 263
  12. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 199
  13. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 194
  14. 2018Mitigating Sybils in Federated Learning Poisoningcited by 161
  15. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injectioncited by 157
  16. 2023Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Modelscited by 148
  17. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 130
  18. 2023Evaluating the Social Impact of Generative AI Systems in Systems and Societycited by 92
  19. 2022DeepSight: Mitigating Backdoor Attacks in Federated Learning Through Deep Model Inspectioncited by 81
  20. 2024Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptationcited by 80
  21. 2022Fairness in Graph Mining: A Surveycited by 76
  22. 2025LLM Agents for Education: Advances and Applicationscited by 59
  23. 2020BaFFLe: Backdoor detection via Feedback-based Federated Learningcited by 53
  24. 2024Persistent Pre-Training Poisoning of LLMscited by 48
  25. 2019Attack-Resistant Federated Learning with Residual-based Reweightingcited by 40
  26. 2023The Philosopher's Stone: Trojaning Plugins of Large Language Modelscited by 37
  27. 2024Denial-of-Service Poisoning Attacks against Large Language Modelscited by 35
  28. 2025Understanding Refusal in Language Models with Sparse Autoencoderscited by 29
  29. 2024Is poisoning a real threat to LLM alignment? Maybe more so than you thinkcited by 28
  30. 2022CrowdGuard: Federated Backdoor Detection in Federated Learningcited by 27
  31. 2023FreqFed: A Frequency Analysis-Based Approach for Mitigating Poisoning Attacks in Federated Learningcited by 27
  32. 2024PEFTGuard: Detecting Backdoor Attacks Against Parameter-Efficient Fine-Tuningcited by 26
  33. 2023Learning and Forgetting Unsafe Examples in Large Language Modelscited by 24
  34. 2024HSF: Defending against Jailbreak Attacks with Hidden State Filteringcited by 23
  35. 2023FedSecurity: Benchmarking Attacks and Defenses in Federated Learning and Federated LLMscited by 21
  36. 2024BadActs: A Universal Backdoor Defense in the Activation Spacecited by 21
  37. 2024The Challenges of Evaluating LLM Applications: An Analysis of Automated, Human, and LLM-Based Approachescited by 18
  38. 2024Emerging Safety Attack and Defense in Federated Instruction Tuning of Large Language Modelscited by 18

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026SafeSeek: Universal Attribution of Safety Circuits in Language Modelscited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2025From Threat to Tool: Leveraging Refusal-Aware Injection Attacks for Safety Alignment
  2. 2024Enhancing Prompt Injection Attacks to LLMs via Poisoning Alignment
  3. 2025An Embarrassingly Simple Defense Against LLM Abliteration Attacks
  4. 2026Deactivating Refusal Triggers: Understanding and Mitigating Overrefusal in Safety Alignment
  5. 2026LLMs Can Unlearn Refusal with Only 1,000 Benign Samples
  6. 2025Revisiting Backdoor Attacks on LLMs: A Stealthy and Practical Poisoning Framework via Harmless Inputs
  7. 2026When Safety Becomes a Vulnerability: Exploiting LLM Alignment Homogeneity for Transferable Blocking in RAG
  8. 2025Jailbreaking LLMs via Semantically Relevant Nested Scenarios with Targeted Toxic Knowledge