mlsanityData to 7 Oct 2026

How Few-shot Demonstrations Affect Prompt-based Defenses Against LLM Jailbreak Attacks

arXiv 2602.04294 ↗First posted 4 Feb 2026cs.CL cs.AI cs.CRcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Large Language Models (LLMs) face increasing threats from jailbreak attacks that bypass safety alignment. While prompt-based defenses such as Role-Oriented Prompts (RoP) and Task-Oriented Prompts (ToP) have shown effectiveness, the role of few-shot demonstrations in these defense strategies remains unclear. Prior work suggests that few-shot examples may compromise safety, but lacks investigation into how few-shot interacts with different system prompt strategies. In this paper, we conduct a comprehensive evaluation on multiple mainstream LLMs across four safety benchmarks (AdvBench, HarmBench, SG-Bench, XSTest) using six jailbreak attack methods. Our key finding reveals that few-shot demonstrations produce opposite effects on RoP and ToP: few-shot enhances RoP's safety rate by up to 4.5% through reinforcing role identity, while it degrades ToP's effectiveness by up to 21.2% through distracting attention from task instructions. Based on these findings, we provide practical recommendations for deploying prompt-based defenses in real-world LLM applications.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 50 of its 83 references that are in the map. First posted on arXiv 4 Feb 2026.

Builds on

50 of 83 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
  2. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  3. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  4. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models1 Sep
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  7. 2023Qwen Technical Reportcited by 3,273
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2020Extracting Training Data from Large Language Modelscited by 2,144
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  11. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  12. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
33 more
  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  2. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  3. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  4. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  5. 2022Red Teaming Language Models with Language Modelscited by 983
  6. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  7. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  8. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
  9. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  10. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  11. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  12. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  13. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  14. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  15. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
  16. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 337
  17. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 280
  18. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  19. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 268
  20. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  21. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  22. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teamingcited by 161
  23. 2024Stealing Part of a Production Language Modelcited by 152
  24. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 150
  25. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 149
  26. 2024Long-form factuality in large language modelscited by 146
  27. 2025SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilitiescited by 133
  28. 2024BadAgent: Inserting and Activating Backdoor Attacks in LLM Agentscited by 106
  29. 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Modelscited by 95
  30. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 68
  31. 2025Pangu Embedded: An Efficient Dual-system LLM Reasoner with Metacognitioncited by 61
  32. 2024SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Typescited by 58
  33. 2024Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interactioncited by 25

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Analysis of LLMs Against Prompt Injection and Jailbreak Attacks
  2. 2025Involuntary Jailbreak: On Self-Prompting Attacks
  3. 2024LLMs Can Defend Themselves Against Jailbreaking in a Practical Manner: A Vision Paper
  4. 2025SecurityLingua: Efficient Defense of LLM Jailbreak Attacks via Security-Aware Prompt Compression
  5. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations
  6. 2025Jailbreaking Attacks vs. Content Safety Filters: How Far Are We in the LLM Safety Arms Race?
  7. 2025PandaGuard: Systematic Evaluation of LLM Safety against Jailbreaking Attacks
  8. 2026Exploring and Developing a Pre-Model Safeguard with Draft Models