mlsanityData to 7 Oct 2026

Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations

arXiv 2310.06387 ↗First posted 10 Oct 2023cs.LG cs.AI cs.CLcited by 401

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Large Language Models (LLMs) have shown remarkable success in various tasks, yet their safety and the risk of generating harmful content remain pressing concerns. In this paper, we delve into the potential of In-Context Learning (ICL) to modulate the alignment of LLMs. Specifically, we propose the In-Context Attack (ICA) which employs harmful demonstrations to subvert LLMs, and the In-Context Defense (ICD) which bolsters model resilience through examples that demonstrate refusal to produce harmful responses. We offer theoretical insights to elucidate how a limited set of in-context demonstrations can pivotally influence the safety alignment of LLMs. Through extensive experiments, we demonstrate the efficacy of ICA and ICD in respectively elevating and mitigating the success rates of jailbreaking prompts. Our findings illuminate the profound influence of ICL on LLM behavior, opening new avenues for improving the safety of LLMs.

Key facts. Cited by 401 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 52 of its 111 references that are in the map. First posted on arXiv 10 Oct 2023.

Builds on

52 of 111 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  2. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  3. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM18 Sep
  4. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models1 Sep
  5. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  7. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun
  8. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug

Other references · 44

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2023Qwen Technical Reportcited by 3,273
  10. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
32 more
  1. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  2. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  3. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  4. 2022Red Teaming Language Models with Language Modelscited by 983
  5. 2022Transformers learn in-context by gradient descentcited by 714
  6. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  7. 2017Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methodscited by 542
  8. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  9. 2022Diffusion Models for Adversarial Purificationcited by 469
  10. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  11. 2023Detecting Language Model Attacks with Perplexitycited by 411
  12. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  13. 2023Low-Resource Languages Jailbreak GPT-4cited by 290
  14. 2023Certifying LLM Safety against Adversarial Promptingcited by 248
  15. 2023Pretraining Language Models with Human Preferencescited by 245
  16. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  17. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  18. 2022Active Example Selection for In-Context Learningcited by 195
  19. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  20. 2017On the (Statistical) Detection of Adversarial Examplescited by 180
  21. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  22. 2022Instruction Induction: From Few Examples to Natural Language Task Descriptionscited by 152
  23. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 152
  24. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 149
  25. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Modelscited by 128
  26. 2023The Learnability of In-Context Learningcited by 126
  27. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 111
  28. 2023Are Emergent Abilities in Large Language Models just In-Context Learning?cited by 109
  29. 2021Bad Characters: Imperceptible NLP Attackscited by 69
  30. 2023Adversarial Demonstration Attacks on Large Language Modelscited by 58
  31. 2023kNN Prompting: Beyond-Context Learning with Calibration-Free Nearest Neighbor Inferencecited by 47
  32. 2023Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreakscited by 45

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 150; 2025: 163; 2026: 74.

401 papers.

Sort:

By area:

  1. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  2. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  3. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  4. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 337
  5. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 255
  6. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
  7. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 222
  8. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  9. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 197
  10. 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMscited by 185
  11. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 179
  12. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
38 more
  1. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  2. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 150
  3. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 149
  4. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 146
  5. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attackscited by 140
  6. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attackscited by 135
  7. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  8. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  9. 2024Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Modelscited by 121
  10. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  11. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performancecited by 111
  12. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  13. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 108
  14. 2023How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMscited by 103
  15. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Promptcited by 103
  16. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Modelscited by 102
  17. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  18. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 94
  19. 2024AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attackscited by 94
  20. 2025H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning Models, Including OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinkingcited by 94
  21. 2024Machine Against the RAG: Jamming Retrieval-Augmented Generation with Blocker Documentscited by 93
  22. 2024How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden Statescited by 92
  23. 2024DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakerscited by 91
  24. 2024Guardrail Baselines for Unlearning in LLMscited by 90
  25. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 89
  26. 2024Detoxifying Large Language Models via Knowledge Editingcited by 86
  27. 2024SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Modelcited by 82
  28. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
  29. 2025Prompt Flow Integrity to Prevent Privilege Escalation in LLM Agentscited by 81
  30. 2023Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinkingcited by 78
  31. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editingcited by 77
  32. 2024A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systemscited by 74
  33. 2024Does Refusal Training in LLMs Generalize to the Past Tense?cited by 74
  34. 2024Safeguarding Large Language Models: A Surveycited by 73
  35. 2024Best-of-N Jailbreakingcited by 71
  36. 2024CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completioncited by 70
  37. 2024Defending Large Language Models against Jailbreak Attacks via Semantic Smoothingcited by 68
  38. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 68

50 most cited of 401 All 401 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Playing the Fool: Jailbreaking LLMs and Multimodal LLMs with Out-of-Distribution Strategy
  2. 2024Jailbreaking Large Language Models Through Alignment Vulnerabilities in Out-of-Distribution Settings
  3. 2025Formalization Driven LLM Prompt Jailbreaking via Reinforcement Learning
  4. 2025When Harmless Words Harm: A New Threat to LLM Safety via Conceptual Triggers
  5. 2026Structural Jailbreaks Generalize but Do Not Compound: A cross-provider and multilingual study of Involuntary In-Context Learning
  6. 2025Do Methods to Jailbreak and Defend LLMs Generalize Across Languages?
  7. 2025Jailbreaking Attacks vs. Content Safety Filters: How Far Are We in the LLM Safety Arms Race?
  8. 2025CCJA: Context-Coherent Jailbreak Attack for Aligned Large Language Models