mlsanityData to 7 Oct 2026

Defending Jailbreak Prompts via In-Context Adversarial Game

arXiv 2402.13148 ↗First posted 20 Feb 2024cs.LG cs.CRcited by 27

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large Language Models (LLMs) demonstrate remarkable capabilities across diverse applications. However, concerns regarding their security, particularly the vulnerability to jailbreak attacks, persist. Drawing inspiration from adversarial training in deep learning and LLM agent learning processes, we introduce the In-Context Adversarial Game (ICAG) for defending against jailbreaks without the need for fine-tuning. ICAG leverages agent learning to conduct an adversarial game, aiming to dynamically extend knowledge to defend against jailbreaks. Unlike traditional methods that rely on static datasets, ICAG employs an iterative process to enhance both the defense and attack agents. This continuous improvement process strengthens defenses against newly generated jailbreak prompts. Our empirical studies affirm ICAG's efficacy, where LLMs safeguarded by ICAG exhibit significantly reduced jailbreak success rates across various attack scenarios. Moreover, ICAG demonstrates remarkable transferability to other LLMs, indicating its potential as a versatile defense mechanism.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 36 of its 65 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

36 of 65 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability13 Feb
  2. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization15 Nov
  3. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation6 Nov
  4. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  5. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  6. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  7. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks5 Oct
  8. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  9. 2023Certifying LLM Safety against Adversarial Prompting6 Sep
  10. 2023Detecting Language Model Attacks with Perplexity27 Aug
  11. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  12. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
1 more
  1. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 23

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2017Billion-scale similarity search with GPUscited by 2,477
  8. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  9. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  10. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  11. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  12. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
11 more
  1. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  2. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  3. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  4. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  5. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 213
  6. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
  7. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 202
  8. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teamingcited by 161
  9. 2023Retroformer: Retrospective Large Language Agents with Policy Gradient Optimizationcited by 110
  10. 2023The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensivenesscited by 63
  11. 2024SceMQA: A Scientific College Entrance Level Multimodal Question Answering Benchmarkcited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 15; 2026: 5.

27 papers.

Sort:

By area:

  1. 2025Agentic Context Engineering: Evolving Contexts for Self-Improving Language Modelscited by 349
  2. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  3. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 68
  4. 2025One Token to Fool LLM-as-a-Judgecited by 56
  5. 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 44
  6. 2024Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Modelscited by 23
  7. 2024Mission Impossible: A Statistical Perspective on Jailbreaking LLMscited by 21
  8. 2025EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental Health Safetycited by 21
  9. 2025Dissecting Logical Reasoning in LLMs: A Fine-Grained Evaluation and Supervision Studycited by 17
  10. 2025Jailbreak Attack Initializations as Extractors of Compliance Directionscited by 10
  11. 2024LoRA-Guard: Parameter-Efficient Guardrail Adaptation for Content Moderation of Large Language Modelscited by 7
  12. 2026Be Your Own Red Teamer: Safety Alignment via Self-Play and Reflective Experience Replaycited by 6
15 more
  1. 2024Enhancing the Capability and Robustness of Large Language Models through Reinforcement Learning-Driven Query Refinementcited by 5
  2. 2024LLMStinger: Jailbreaking LLMs using RL fine-tuned LLMscited by 5
  3. 2025Active Attacks: Red-teaming LLMs via Adaptive Environmentscited by 5
  4. 2025SafeInt: Shielding Large Language Models from Jailbreak Attacks via Safety-Aware Representation Interventioncited by 4
  5. 2025Breach in the Shield: Unveiling the Vulnerabilities of Large Language Modelscited by 4
  6. 2025JavelinGuard: Low-Cost Transformer Architectures for LLM Securitycited by 4
  7. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 4
  8. 2025Adversarial Attack-Defense Co-Evolution for LLM Safety Alignment via Tree-Group Dual-Aware Search and Optimizationcited by 4
  9. 2026Distribution-Aware Reward Estimation for Test-Time Reinforcement Learningcited by 2
  10. 2025Enhancing the Safety of Medical Vision-Language Models by Synthetic Demonstrationscited by 1
  11. 2026Profit is the Red Team: Stress-Testing Agents in Strategic Economic Interactionscited by 1
  12. 2025TombRaider: Entering the Vault of History to Jailbreak Large Language Modelscited by 0
  13. 2025TuCo: Measuring the Contribution of Fine-Tuning to Individual Responses of LLMscited by 0
  14. 2026Dictionary-Aligned Concept Control for Safeguarding Multimodal LLMscited by 0
  15. 2026Model-Agnostic Lifelong LLM Safety via Externalized Attack-Defense Co-Evolutioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Jailbreak Mimicry: Automated Discovery of Narrative-Based Jailbreaks for Large Language Models
  2. 2026Jailbreaking Leaves a Trace: Understanding and Detecting Jailbreak Attacks from Internal Representations of Large Language Models
  3. 2025Adversarial Reasoning at Jailbreaking Time
  4. 2025Defense-to-Attack: Bypassing Weak Defenses Enables Stronger Jailbreaks in Vision-Language Models
  5. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Survey
  6. 2026The Art of the Jailbreak: Formulating Jailbreak Attacks for LLM Security Beyond Binary Scoring
  7. 2025Test-Time Immunization: A Universal Defense Framework Against Jailbreaks for (Multimodal) Large Language Models
  8. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing