mlsanityData to 14 Oct 2026

Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models

arXiv 2310.02949 ↗First posted 4 Oct 2023cs.CL cs.AI cs.CRcited by 297

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

Warning: This paper contains examples of harmful language, and reader discretion is recommended. The increasing open release of powerful large language models (LLMs) has facilitated the development of downstream applications by reducing the essential cost of data annotation and computation. To ensure AI safety, extensive safety-alignment measures have been conducted to armor these models against malicious use (primarily hard prompt attack). However, beneath the seemingly resilient facade of the armor, there might lurk a shadow. By simply tuning on 100 malicious examples with 1 GPU hour, these safely aligned LLMs can be easily subverted to generate harmful content. Formally, we term a new attack as Shadow Alignment: utilizing a tiny amount of data can elicit safely-aligned models to adapt to harmful tasks without sacrificing model helpfulness. Remarkably, the subverted models retain their capability to respond appropriately to regular inquiries. Experiments across 8 models released by 5 different organizations (LLaMa-2, Falcon, InternLM, BaiChuan2, Vicuna) demonstrate the effectiveness of shadow alignment attack. Besides, the single-turn English-only attack successfully transfers to multi-turn dialogue and other languages. This study serves as a clarion call for a collective effort to overhaul and fortify the safety of open-source LLMs against malicious attackers.

Key facts. Cited by 297 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 49 references that are in the map. First posted on arXiv 4 Oct 2023.

Builds on

40 of 49 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM18 Sep
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  3. 2023Are aligned neural networks adversarially aligned?26 Jun

Other references · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
25 more
  1. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  2. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  3. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  4. 2023LIMA: Less Is More for Alignmentcited by 1,100
  5. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  6. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  7. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  8. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  9. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  10. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 488
  11. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 422
  12. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  13. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 223
  14. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 199
  15. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 157
  16. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 155
  17. 2023On the Exploitability of Instruction Tuningcited by 116
  18. 2022Self-Destructing Models: Increasing the Costs of Harmful Dual Uses of Foundation Modelscited by 57
  19. 2022SafeText: A Benchmark for Exploring Physical Safety in Language Modelscited by 41
  20. 2023A Preliminary Study of the Intrinsic Relationship between Complexity and Alignmentcited by 33
  21. 2023OpinionGPT: Modelling Explicit Biases in Instruction-Tuned LLMscited by 29
  22. 2023DoG-Instruct: Towards Premium Instruction-Tuning Data via Text-Grounded Instruction Wrappingcited by 16
  23. 2023The Poison of Alignmentcited by 10
  24. 2023Text Alignment Is An Efficient Unified Model for Massive NLP Taskscited by 8
  25. 2023Balancing Transparency and Risk: The Security and Privacy Risks of Open-Source Machine Learning Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 15; 2024: 117; 2025: 83; 2026: 82.

297 papers.

Sort:

By area:

  1. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  2. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
  3. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  4. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
  5. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 288
  6. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 274
  7. 2024Rethinking Machine Unlearning for Large Language Modelscited by 257
  8. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
  9. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 230
  10. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 195
  11. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 194
  12. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 185
38 more
  1. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 175
  2. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 173
  3. 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 171
  4. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 163
  5. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 153
  6. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 138
  7. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Securitycited by 134
  8. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 130
  9. 2024LLM Agents can Autonomously Exploit One-day Vulnerabilitiescited by 128
  10. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 126
  11. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 125
  12. 2024An Adversarial Perspective on Machine Unlearning for AI Safetycited by 123
  13. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 118
  14. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  15. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 115
  16. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 115
  17. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 112
  18. 2024LLM Agents can Autonomously Hack Websitescited by 109
  19. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 107
  20. 2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuningcited by 96
  21. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 96
  22. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 96
  23. 2024Teams of LLM Agents can Exploit Zero-Day Vulnerabilitiescited by 95
  24. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 95
  25. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
  26. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 92
  27. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  28. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 90
  29. 2024Coercing LLMs to do and reveal (almost) anythingcited by 89
  30. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 88
  31. 2024ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stagescited by 82
  32. 2024Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptationcited by 80
  33. 2024CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Modelscited by 76
  34. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuningcited by 72
  35. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 71
  36. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 70
  37. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Trainingcited by 69
  38. 2024On the Societal Impact of Open Foundation Modelscited by 65

50 most cited of 297 All 297 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Emulated Disalignment: Safety Alignment for Large Language Models May Backfire!
  2. 2023Large Language Model Alignment: A Survey
  3. 2025Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models?
  4. 2023Bergeron: Combating Adversarial Attacks through a Conscience-Based Alignment Framework
  5. 2024Towards Comprehensive Post Safety Alignment of Large Language Models via Safety Patching
  6. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  7. 2025Safety Alignment Depth in Large Language Models: A Markov Chain Perspective
  8. 2024Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activations