mlsanityData to 7 Oct 2026

No Free Lunch for Defending Against Prefilling Attack by In-Context Learning

arXiv 2412.12192 ↗First posted 13 Dec 2024cs.CR cs.AIcited by 2

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

The security of Large Language Models (LLMs) has become an important research topic since the emergence of ChatGPT. Though there have been various effective methods to defend against jailbreak attacks, prefilling attacks remain an unsolved and popular threat against open-sourced LLMs. In-Context Learning (ICL) offers a computationally efficient defense against various jailbreak attacks, yet no effective ICL methods have been developed to counter prefilling attacks. In this paper, we: (1) show that ICL can effectively defend against prefilling jailbreak attacks by employing adversative sentence structures within demonstrations; (2) characterize the effectiveness of this defense through the lens of model size, number of demonstrations, over-defense, integration with other jailbreak attacks, and the presence of safety alignment. Given the experimental results and our analysis, we conclude that there is no free lunch for defending against prefilling jailbreak attacks with ICL. On the one hand, current safety alignment methods fail to mitigate prefilling jailbreak attacks, but adversative structures within ICL demonstrations provide robust defense across various model sizes and complex jailbreak attacks. On the other hand, LLMs exhibit similar over-defensiveness when utilizing ICL demonstrations with adversative structures, and this behavior appears to be independent of model size.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 26 of its 33 references that are in the map. First posted on arXiv 13 Dec 2024.

Builds on

26 of 33 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  2. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  3. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  4. 2023Detecting Language Model Attacks with Perplexity27 Aug
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  6. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 20

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  2. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  3. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  4. 2023LIMA: Less Is More for Alignmentcited by 1,080
  5. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  6. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  7. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  8. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  9. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  10. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 224
  11. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  12. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 158
8 more
  1. 2022Ask Me Anything: A simple strategy for prompting language modelscited by 155
  2. 2023Compositional Exemplars for In-context Learningcited by 143
  3. 2023An LLM can Fool Itself: A Prompt-Based Adversarial Attackcited by 111
  4. 2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
  5. 2023The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensivenesscited by 63
  6. 2024Intrinsic Self-correction for Enhanced Morality: An Analysis of Internal Mechanisms and the Superficial Hypothesiscited by 15
  7. 2024A Survey to Recent Progress Towards Understanding In-Context Learningcited by 12
  8. 2024Self-correction is Not An Innate Capability in Language Modelscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Harmonizing Multi-Objective LLM Unlearning via Unified Domain Representation and Bidirectional Logit Distillationcited by 0
  2. 2026Understanding In-Context Multimodal Jailbreaks via Posterior Reweightingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Prefill-level Jailbreak: A Black-Box Risk Analysis of Large Language Models
  2. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Survey
  3. 2025Scalable Defense against In-the-wild Jailbreaking Attacks with Safety Context Retrieval
  4. 2024textitMMJ-Bench: A Comprehensive Study on Jailbreak Attacks and Defenses for Multimodal Large Language Models
  5. 2026Defending Large Language Models Against Jailbreak Attacks via In-Decoding Safety-Awareness Probing
  6. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing
  7. 2025SafeLLM: Unlearning Harmful Outputs from Large Language Models against Jailbreak Attacks
  8. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignment