mlsanityData to 14 Oct 2026

What Really Matters in Many-Shot Attacks? An Empirical Study of Long-Context Vulnerabilities in LLMs

arXiv 2505.19773 ↗First posted 26 May 2025cs.CL cs.CRcited by 8

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

We investigate long-context vulnerabilities in Large Language Models (LLMs) through Many-Shot Jailbreaking (MSJ). Our experiments utilize context length of up to 128K tokens. Through comprehensive analysis with various many-shot attack settings with different instruction styles, shot density, topic, and format, we reveal that context length is the primary factor determining attack effectiveness. Critically, we find that successful attacks do not require carefully crafted harmful content. Even repetitive shots or random dummy text can circumvent model safety measures, suggesting fundamental limitations in long-context processing capabilities of LLMs. The safety behavior of well-aligned models becomes increasingly inconsistent with longer contexts. These findings highlight significant safety gaps in context expansion capabilities of LLMs, emphasizing the need for new safety mechanisms.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 40 of its 63 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

40 of 63 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defenses3 Jun
  2. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  3. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  4. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM18 Sep
  5. 2023Detecting Language Model Attacks with Perplexity27 Aug
  6. 2023Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks24 May

Other references · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  9. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  10. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  11. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  12. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
22 more
  1. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  2. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  3. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  4. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  5. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  6. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 360
  7. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
  8. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  9. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 223
  10. 2022Predictability and Surprise in Large Generative Modelscited by 210
  11. 2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 139
  12. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 133
  13. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 128
  14. 2023Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineeringcited by 94
  15. 2023Better Zero-Shot Reasoning with Self-Adaptive Promptingcited by 52
  16. 2024LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answeringcited by 49
  17. 2023In-Context Learning with Many Demonstration Examplescited by 39
  18. 2024SPML: A DSL for Defending Language Models Against Prompt Attackscited by 34
  19. 2024Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignmentcited by 31
  20. 2024How (un)ethical are instruction-centric responses of LLMs? Unveiling the vulnerabilities of safety guardrails to harmful queriescited by 27
  21. 2021Long Context Question Answering via Supervised Contrastive Learningcited by 17
  22. 2023LifeTox: Unveiling Implicit Toxicity in Life Advicecited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026AC4A: Access Control for Agentscited by 5
  2. 2026The Defense Trilemma: Why Prompt Injection Defense Wrappers Fail?cited by 2
  3. 2026ChartAttack: Testing the Vulnerability of LLMs to Malicious Prompting in Chart Generationcited by 0
  4. 2026Mitigating Many-shot Jailbreak Attacks with One Single Demonstrationcited by 0
  5. 2026Safety in Batches? Understanding and Mitigating Safety Failures in Batch Promptingcited by 0
  6. 2026The Safeguard Worked. Is the LLM System Safer?cited by 0
  7. 2026Constitutional adapters: Inference-time interventions for misalignment and misusecited by 0
  8. 2026The Geometry of Harmfulness in Multi-Turn Attackscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Models
  2. 2025You Can't Eat Your Cake and Have It Too: The Performance Degradation of LLMs with Jailbreak Defense
  3. 2025Evolving Security in LLMs: A Study of Jailbreak Attacks and Defenses
  4. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMs
  5. 2024Harnessing Task Overload for Scalable Jailbreak Attacks on Large Language Models
  6. 2025Beyond Jailbreaks: Revealing Stealthier and Broader LLM Security Risks Stemming from Alignment Failures
  7. 2024Characterizing and Evaluating the Reliability of LLMs against Jailbreak Attacks
  8. 2024Revisiting Jailbreaking for Large Language Models: A Representation Engineering Perspective