mlsanityData to 7 Oct 2026

PANDAS: Improving Many-shot Jailbreaking via Positive Affirmation, Negative Demonstration, and Adaptive Sampling

arXiv 2502.01925 ↗First posted 4 Feb 2025cs.CL cs.CR cs.LGcited by 6

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Many-shot jailbreaking circumvents the safety alignment of LLMs by exploiting their ability to process long input sequences. To achieve this, the malicious target prompt is prefixed with hundreds of fabricated conversational exchanges between the user and the model. These exchanges are randomly sampled from a pool of unsafe question-answer pairs, making it appear as though the model has already complied with harmful instructions. In this paper, we present PANDAS: a hybrid technique that improves many-shot jailbreaking by modifying these fabricated dialogues with Positive Affirmations, Negative Demonstrations, and an optimized Adaptive Sampling method tailored to the target prompt's topic. We also introduce ManyHarm, a dataset of harmful question-answer pairs, and demonstrate through extensive experiments that PANDAS significantly outperforms baseline methods in long-context scenarios. Through attention analysis, we provide insights into how long-context vulnerabilities are exploited and show how PANDAS further improves upon many-shot jailbreaking.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 41 of its 53 references that are in the map. First posted on arXiv 4 Feb 2025.

Builds on

41 of 53 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  2. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models1 Sep
  3. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug

Other references · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  10. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  11. 2024Qwen2 Technical Reportcited by 2,076
  12. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
26 more
  1. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  2. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  3. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  4. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  5. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  6. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  7. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  8. 2022Red Teaming Language Models with Language Modelscited by 983
  9. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  10. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  11. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  12. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  13. 2023What Makes Good Examples for Visual In-Context Learning?cited by 150
  14. 2021Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020cited by 146
  15. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 133
  16. 2023Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMscited by 82
  17. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 68
  18. 2023How Many Demonstrations Do You Need for In-context Learning?cited by 52
  19. 2023On the Role of Attention in Prompt-tuningcited by 49
  20. 2023Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreakscited by 45
  21. 2024In-Context Principle Learning from Mistakescited by 42
  22. 2024Distract Large Language Models for Automatic Jailbreak Attackcited by 40
  23. 2023Attention-Enhancing Backdoor Attacks Against BERT-based Modelscited by 32
  24. 2024Customizing Language Model Responses with Contrastive In-Context Learningcited by 19
  25. 2024Exploring Context Window of Large Language Models via Decomposed Positional Vectorscited by 18
  26. 2024Large Language Models Are Involuntary Truth-Tellers: Exploiting Fallacy Failure for Jailbreak Attackscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025AutoRAN: Automated Hijacking of Safety Reasoning in Large Reasoning Modelscited by 7
  2. 2025Adversarial Attack-Defense Co-Evolution for LLM Safety Alignment via Tree-Group Dual-Aware Search and Optimizationcited by 4
  3. 2026Large Language Models Generate Harmful Responses Using a Distinct Mechanism, Shared Across Harm Typescited by 3
  4. 2025MetaCipher: A Time-Persistent and Universal Multi-Agent Framework for Cipher-Based Jailbreak Attacks for LLMscited by 1
  5. 2026Mitigating Many-shot Jailbreak Attacks with One Single Demonstrationcited by 0
  6. 2026What Do Safety-Aligned LLMs Learn From Mixed Compliance Demonstrations?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reasoned Safety Alignment: Ensuring Jailbreak Defense via Answer-Then-Check
  2. 2024Understanding Jailbreak Success: A Study of Latent Space Dynamics in Large Language Models
  3. 2026LLM-VA: Resolving the Jailbreak-Overrefusal Trade-off via Vector Alignment
  4. 2025Automating Deception: Scalable Multi-Turn LLM Jailbreaks
  5. 2025Attack via Overfitting: 10-shot Benign Fine-tuning to Jailbreak LLMs
  6. 2025PandaGuard: Systematic Evaluation of LLM Safety against Jailbreaking Attacks
  7. 2026MultiBreak: A Scalable and Diverse Multi-turn Jailbreak Benchmark for Evaluating LLM Safety
  8. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Survey