# Jailbreaking in the Haystack

arXiv: https://arxiv.org/abs/2511.04707 · HTML: https://mlsanity.com/paper/2511.04707/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 18 of its 24 references that are in the map.
- First posted on arXiv 5 Nov 2025.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)
- LLM safety and security / [Jailbreaking](https://mlsanity.com/safety/jailbreaking/)

## Builds on (most cited first, 18 of 18)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models](https://mlsanity.com/paper/2310.04451/) (2023)
- [Tree of Attacks: Jailbreaking Black-Box LLMs Automatically](https://mlsanity.com/paper/2312.02119/) (2023)
- [Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack](https://mlsanity.com/paper/2404.01833/) (2024)
- [Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization](https://mlsanity.com/paper/2311.09096/) (2023)
- [Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks](https://mlsanity.com/paper/2401.17263/) (2024)
- [SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents](https://mlsanity.com/paper/2506.15740/) (2025)
- [Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents](https://mlsanity.com/paper/2410.13886/) (2024)
- [On the Emergence of Position Bias in Transformers](https://mlsanity.com/paper/2502.01951/) (2025)
- [FuzzLLM: A Novel and Universal Fuzzing Framework for Proactively Discovering Jailbreak Vulnerabilities in Large Language Models](https://mlsanity.com/paper/2309.05274/) (2023)
- [Red Teaming the Mind of the Machine: A Systematic Evaluation of Prompt Injection and Jailbreak Vulnerabilities in LLMs](https://mlsanity.com/paper/2505.04806/) (2025)
- [Cognitive Overload Attack:Prompt Injection for Long Context](https://mlsanity.com/paper/2410.11272/) (2024)
- [LongSafety: Evaluating Long-Context Safety of Large Language Models](https://mlsanity.com/paper/2502.16971/) (2025)
- [Single-pass Detection of Jailbreaking Input in Large Language Models](https://mlsanity.com/paper/2502.15435/) (2025)
- [PandaGuard: Systematic Evaluation of LLM Safety against Jailbreaking Attacks](https://mlsanity.com/paper/2505.13862/) (2025)

## Built on by (most cited first, 2 of 2)

- [Statistical Estimation of Adversarial Risk in Large Language Models under Best-of-N Sampling](https://mlsanity.com/paper/2601.22636/) (2026)
- [Mitigating Many-shot Jailbreak Attacks with One Single Demonstration](https://mlsanity.com/paper/2605.08277/) (2026)

## Abstract (arXiv)

Recent advances in long-context language models (LMs) have enabled million-token inputs, expanding their capabilities across complex tasks like computer-use agents. Yet, the safety implications of these extended contexts remain unclear. To bridge this gap, we introduce NINJA (short for Needle-in-haystack jailbreak attack), a method that jailbreaks aligned LMs by appending benign, model-generated content to harmful user goals. Critical to our method is the observation that the position of harmful goals play an important role in safety. Experiments on standard safety benchmark, HarmBench, show that NINJA significantly increases attack success rates across state-of-the-art open and proprietary models, including LLaMA, Qwen, Mistral, and Gemini. Unlike prior jailbreaking methods, our approach is low-resource, transferable, and less detectable. Moreover, we show that NINJA is compute-optimal -- under a fixed compute budget, increasing context length can outperform increasing the number of trials in best-of-N jailbreak. These findings reveal that even benign long contexts -- when crafted with careful goal positioning -- introduce fundamental vulnerabilities in modern LMs.