mlsanityData to 14 Oct 2026

Mitigating Many-shot Jailbreak Attacks with One Single Demonstration

arXiv 2605.08277 ↗First posted 8 May 2026cs.CR cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Many-shot jailbreaking (MSJ) causes safety-aligned language models to answer harmful queries by preceding them with many harmful question-answer demonstrations. We study why this attack becomes stronger as the number of demonstrations increases. Empirically, we find that MSJ induces a progressive activation drift: the representation of a fixed harmful query moves step by step away from the safety-aligned region as more harmful demonstrations are added. Theoretically, we show that this drift can be interpreted as implicit malicious fine-tuning: conditioning on N harmful demonstrations induces SGD-style updates equivalent to optimizing on the corresponding N harmful samples. This view turns the attack mechanism into a defense principle. We append a fixed one-shot safety demonstration at inference time, which induces a counteracting safety-oriented update and restores refusal behavior. The resulting method improves the model's robustness to MSJ without modifying its parameters or requiring white-box access at deployment. Code is available at https://github.com/Thecommonirin/SafeEnd.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 47 of its 56 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

47 of 56 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance5 Jan
  2. 2025MTSA: Multi-turn Safety Alignment for LLMs through Multi-round Red-teaming22 May
  3. 2025PANDAS: Improving Many-shot Jailbreaking via Positive Affirmation, Negative Demonstration, and Adaptive Sampling4 Feb
  4. 2024SecAlign: Defending Against Prompt Injection with Preference Optimization7 Oct
  5. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  6. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models1 Sep
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 40

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  9. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  10. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  11. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  12. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
28 more
  1. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  2. 2025OpenAI GPT-5 System Cardcited by 989
  3. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  4. 2022Transformers learn in-context by gradient descentcited by 732
  5. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  6. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  7. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
  8. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 390
  9. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  10. 2025Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learningcited by 161
  11. 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agentscited by 125
  12. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 122
  13. 2025Foot-In-The-Door: A Multi-turn Jailbreak for LLMscited by 54
  14. 2024Navigating the OverKill in Large Language Modelscited by 44
  15. 2025Learning without training: The implicit dynamics of in-context learningcited by 43
  16. 2024Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Modelscited by 23
  17. 2023Do pretrained Transformers Learn In-Context by Gradient Descent?cited by 18
  18. 2026SEMA: Simple yet Effective Learning for Multi-Turn Jailbreak Attackscited by 12
  19. 2026Internal Safety Collapse in Frontier Large Language Modelscited by 12
  20. 2025A Representation Engineering Perspective on the Effectiveness of Multi-Turn Jailbreakscited by 10
  21. 2024Jailbreaking? One Step Is Enough!cited by 9
  22. 2025What Really Matters in Many-Shot Attacks? An Empirical Study of Long-Context Vulnerabilities in LLMscited by 8
  23. 2026COLD-Steer: Steering Large Language Models via In-Context One-step Learning Dynamicscited by 8
  24. 2025Activation Approximations Can Incur Safety Vulnerabilities Even in Aligned LLMs: Comprehensive Analysis and Defensecited by 7
  25. 2026Knowledge-Driven Multi-Turn Jailbreaking on Large Language Modelscited by 7
  26. 2026Demystifying the Slash Pattern in Attention: The Role of RoPEcited by 4
  27. 2025Many-Turn Jailbreakingcited by 2
  28. 2025Jailbreaking in the Haystackcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Reasoned Safety Alignment: Ensuring Jailbreak Defense via Answer-Then-Check
  2. 2025Mitigating Many-Shot Jailbreaking
  3. 2025SafeInt: Shielding Large Language Models from Jailbreak Attacks via Safety-Aware Representation Intervention
  4. 2025Beyond Jailbreaks: Revealing Stealthier and Broader LLM Security Risks Stemming from Alignment Failures
  5. 2025GRAF: Multi-turn Jailbreaking via Global Refinement and Active Fabrication
  6. 2026Defending Large Language Models Against Jailbreak Attacks via In-Decoding Safety-Awareness Probing
  7. 2025Fine-Tuning Jailbreaks under Highly Constrained Black-Box Settings: A Three-Pronged Approach
  8. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding