mlsanityData to 7 Oct 2026

The Poison of Alignment

arXiv 2308.13449 ↗First posted 25 Aug 2023cs.CLcited by 10

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

From the perspective of content safety issues, alignment has shown to limit large language models' (LLMs) harmful content generation. This intentional method of reinforcing models to not respond to certain user inputs seem to be present in many modern open-source instruction tuning datasets such as OpenAssistant or Guanaco. We introduce a novel insight to an instruction-tuned model's performance affected by the presence of alignment in supervised fine-tuning dataset. To be specific, we noticed that alignment acts as if it is poisoning the instruction dataset. Experimentally, we demonstrate that aligned answers significantly worsen the performance of the resulting fine-tuned model's on various reasoning benchmarks such as Big Bench (BBH), Massive Multitask Language Understanding (MMLU), Human Eval, and Discrete Reasoning Over Paragraphs (DROP), performing worse than the counterpart tuned without alignment by 4-33%.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 23 of its 28 references that are in the map. First posted on arXiv 25 Aug 2023.

Builds on

23 of 28 references.

References · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  11. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  12. 2023LIMA: Less Is More for Alignmentcited by 1,080
11 more
  1. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  2. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  3. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  4. 2020Aligning AI With Shared Human Valuescited by 706
  5. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  6. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  7. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  8. 2023Poisoning Language Models During Instruction Tuningcited by 254
  9. 2023The False Promise of Imitating Proprietary LLMscited by 254
  10. 2023On the Exploitability of Instruction Tuningcited by 110
  11. 2023INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large Language Modelscited by 62

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 2; 2026: 1.

10 papers.

  1. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  2. 2025STAR-1: Safer Alignment of Reasoning LLMs with 1K Datacited by 67
  3. 2024Constructing Benchmarks and Interventions for Combating Hallucinations in LLMscited by 19
  4. 2024PrimeGuard: Safe and Helpful LLMs through Tuning-Free Routingcited by 10
  5. 2024A Fine-tuning Enhanced RAG System with Quantized Influence Measure as AI Judgecited by 9
  6. 2024Would I Lie To You? Inference Time Alignment of Language Models using Direct Preference Headscited by 5
  7. 2024Narrative-of-Thought: Improving Temporal Reasoning of Large Language Models via Recounted Narrativescited by 5
  8. 2026Understanding LLM Behavior When Encountering User-Supplied Harmful Content in Harmless Taskscited by 3
  9. 2025Listener-Rewarded Thinking in VLMs for Image Preferencescited by 2
  10. 2024Nevermind: Instruction Override and Moderation in Large Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Picky LLMs and Unreliable RMs: An Empirical Study on Safety Alignment after Instruction Tuning
  2. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models
  3. 2024Do as I do (Safely): Mitigating Task-Specific Fine-tuning Risks in Large Language Models
  4. 2023Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysis
  5. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!
  6. 2025Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality
  7. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning
  8. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning