mlsanityData to 7 Oct 2026

Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defenses

arXiv 2406.01288 ↗First posted 3 Jun 2024cs.CL cs.AI cs.CRcited by 68

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Recently, Anil et al. (2024) show that many-shot (up to hundreds of) demonstrations can jailbreak state-of-the-art LLMs by exploiting their long-context capability. Nevertheless, is it possible to use few-shot demonstrations to efficiently jailbreak LLMs within limited context sizes? While the vanilla few-shot jailbreaking may be inefficient, we propose improved techniques such as injecting special system tokens like [/INST] and employing demo-level random search from a collected demo pool. These simple techniques result in surprisingly effective jailbreaking against aligned LLMs (even with advanced defenses). For examples, our method achieves >80% (mostly >95%) ASRs on Llama-2-7B and Llama-3-8B without multiple restarts, even if the models are enhanced by strong defenses such as perplexity detection and/or SmoothLLM, which is challenging for suffix-based jailbreaking. In addition, we conduct comprehensive and elaborate (e.g., making sure to use correct system prompts) evaluations against other aligned LLMs and advanced defenses, where our method consistently achieves nearly 100% ASRs. Our code is available at https://github.com/sail-sg/I-FSJ.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 64 of its 74 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

64 of 74 references.

Closest ancestors · 34

Same direction, newest first

  1. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs21 Apr
  2. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs11 Apr
  3. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks2 Apr
  4. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models28 Mar
  5. 2024RigorLLM: Resilient Guardrails for Large Language Models against Undesired Content19 Mar
  6. 2024Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes1 Mar
  7. 2024Defending LLMs against Jailbreaking Attacks via Backtranslation26 Feb
  8. 2024Defending Large Language Models against Jailbreak Attacks via Semantic Smoothing25 Feb
  9. 2024PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-Rails24 Feb
  10. 2024Defending Jailbreak Prompts via In-Context Adversarial Game20 Feb
  11. 2024Query-Based Adversarial Prompt Generation19 Feb
  12. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding14 Feb
22 more
  1. 2024Fight Back Against Jailbreaking via Prompt Adversarial Tuning9 Feb
  2. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks30 Jan
  3. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan
  4. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  5. 2023Token-Level Adversarial Prompt Detection Based on Perplexity Measures and Contextual Information20 Nov
  6. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization15 Nov
  7. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreaker6 Nov
  8. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  9. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  10. 2023Multilingual Jailbreak Challenges in Large Language Models10 Oct
  11. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  12. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks5 Oct
  13. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  14. 2023Low-Resource Languages Jailbreak GPT-43 Oct
  15. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM18 Sep
  16. 2023Certifying LLM Safety against Adversarial Prompting6 Sep
  17. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Models4 Sep
  18. 2023Detecting Language Model Attacks with Perplexity27 Aug
  19. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  20. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  21. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  22. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  8. 2023Qwen Technical Reportcited by 3,273
  9. 2023Mistral 7Bcited by 3,065
  10. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  11. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  12. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
18 more
  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  2. 2022Red Teaming Language Models with Language Modelscited by 983
  3. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  4. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  5. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
  6. 2023Are aligned neural networks adversarially aligned?cited by 305
  7. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  8. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  9. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  10. 2023Pretraining Language Models with Human Preferencescited by 245
  11. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  12. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 183
  13. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 149
  14. 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Gamecited by 135
  15. 2023Evil Geniuses: Delving into the Safety of LLM-based Agentscited by 103
  16. 2024PAL: Proxy-Guided Black-Box Attack on Large Language Modelscited by 46
  17. 2023BadLlama: cheaply removing safety fine-tuning from Llama 2-Chat 13Bcited by 35
  18. 2024SPML: A DSL for Defending Language Models Against Prompt Attackscited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 37; 2026: 16.

68 papers.

Sort:

By area:

  1. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  2. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 73
  3. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  4. 2024Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchycited by 62
  5. 2024SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Mannercited by 57
  6. 2024Jailbreak Large Vision-Language Models Through Multi-Modal Linkagecited by 53
  7. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMscited by 47
  8. 2024JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Frameworkcited by 44
  9. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 42
  10. 2025Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safetycited by 38
  11. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 31
  12. 2024AdvPrefix: An Objective for Nuanced LLM Jailbreakscited by 25
38 more
  1. 2024Adversarial Tuning: Defending Against Jailbreak Attacks for LLMscited by 23
  2. 2025RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Modelscited by 23
  3. 2025Lifelong Safety Alignment for Language Modelscited by 13
  4. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  5. 2025PIG: Privacy Jailbreak Attack on LLMs via Gradient-based Iterative In-Context Optimizationcited by 10
  6. 2025MixAT: Combining Continuous and Discrete Adversarial Training for LLMscited by 10
  7. 2026Obscure but Effective: Classical Chinese Jailbreak Prompt Optimization via Bio-Inspired Searchcited by 10
  8. 2025GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methodscited by 9
  9. 2025Unintended Misalignment from Agentic Fine-Tuning: Risks and Mitigationcited by 9
  10. 2025Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoringcited by 9
  11. 2024Stepwise Reasoning Error Disruption Attack of LLMscited by 8
  12. 2025Evolving Security in LLMs: A Study of Jailbreak Attacks and Defensescited by 8
  13. 2025Towards the Worst-case Robustness of Large Language Modelscited by 7
  14. 2025What Really Matters in Many-Shot Attacks? An Empirical Study of Long-Context Vulnerabilities in LLMscited by 7
  15. 2025PANDAS: Improving Many-shot Jailbreaking via Positive Affirmation, Negative Demonstration, and Adaptive Samplingcited by 6
  16. 2025EVOREFUSE: Evolutionary Prompt Optimization for Evaluation and Mitigation of LLM Over-Refusal to Pseudo-Malicious Instructionscited by 6
  17. 2025When Style Breaks Safety: Defending LLMs Against Superficial Style Alignmentcited by 6
  18. 2025NEXUS: Network Exploration for eXploiting Unsafe Sequences in Multi-Turn LLM Jailbreakscited by 6
  19. 2025Chain-of-Lure: A Universal Jailbreak Attack Framework using Unconstrained Synthetic Narrativescited by 5
  20. 2024Plentiful Jailbreaks with String Compositionscited by 4
  21. 2025Exploiting Instruction-Following Retrievers for Malicious Information Retrievalcited by 4
  22. 2025Mitigating Jailbreaks with Intent-Aware LLMscited by 4
  23. 2025SafeLLM: Unlearning Harmful Outputs from Large Language Models against Jailbreak Attackscited by 4
  24. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 4
  25. 2025Adversarial Attack-Defense Co-Evolution for LLM Safety Alignment via Tree-Group Dual-Aware Search and Optimizationcited by 4
  26. 2026Defending Large Language Models Against Jailbreak Attacks via In-Decoding Safety-Awareness Probingcited by 4
  27. 2025Imperceptible Jailbreaking against Large Language Modelscited by 3
  28. 2025Phonetic Perturbations Reveal Tokenizer-Rooted Safety Gaps in LLMscited by 2
  29. 2025Many-Turn Jailbreakingcited by 2
  30. 2026Unsafe by Reciprocity: How Generation-Understanding Coupling Undermines Safety in Unified Multimodal Modelscited by 2
  31. 2024Desert Camels and Oil Sheikhs: Arab-Centric Red Teaming of Frontier LLMscited by 1
  32. 2025Self-Instruct Few-Shot Jailbreaking: Decompose the Attack into Pattern and Behavior Learningcited by 1
  33. 2025MAD-MAX: Modular And Diverse Malicious Attack MiXtures for Automated LLM Red Teamingcited by 1
  34. 2025Jailbreak Detection in Clinical Training LLMs Using Feature-Based Predictive Modelscited by 1
  35. 2025SoK: A Comprehensive Security Analysis of Jailbreak Resilience in GPT and DeepSeek Modelscited by 1
  36. 2025Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generationcited by 1
  37. 2025One Leak Away: How Pretrained Model Exposure Amplifies Jailbreak Risks in Finetuned LLMscited by 1
  38. 2026Talk is (Not) Cheap: A Taxonomy and Benchmark Coverage Audit for LLM Attackscited by 1

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attacks
  2. 2024JailPO: A Novel Black-box Jailbreak Framework via Preference Optimization against Aligned LLMs
  3. 2024Efficient LLM-Jailbreaking via Multimodal-LLM Jailbreak
  4. 2026Break Me If You Can: Self-Jailbreaking of Aligned LLMs via Lexical Insertion Prompting
  5. 2024No Free Lunch for Defending Against Prefilling Attack by In-Context Learning
  6. 2024Divide and Conquer: A Hybrid Strategy Defeats Multimodal Large Language Models
  7. 2025Test-Time Immunization: A Universal Defense Framework Against Jailbreaks for (Multimodal) Large Language Models
  8. 2024AttnGCG: Enhancing Jailbreaking Attacks on LLMs with Attention Manipulation