mlsanityData to 7 Oct 2026

Jailbreak Defense in a Narrow Domain: Limitations of Existing Methods and a New Transcript-Classifier Approach

arXiv 2412.02159 ↗First posted 3 Dec 2024cs.LG cs.AI cs.CLcited by 6

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Defending large language models against jailbreaks so that they never engage in a broadly-defined set of forbidden behaviors is an open problem. In this paper, we investigate the difficulty of jailbreak-defense when we only want to forbid a narrowly-defined set of behaviors. As a case study, we focus on preventing an LLM from helping a user make a bomb. We find that popular defenses such as safety training, adversarial training, and input/output classifiers are unable to fully solve this problem. In pursuit of a better solution, we develop a transcript-classifier defense which outperforms the baseline defenses we test. However, our classifier defense still fails in some circumstances, which highlights the difficulty of jailbreak-defense even in a narrow domain.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 18 of its 35 references that are in the map. First posted on arXiv 3 Dec 2024.

Builds on

18 of 35 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet27 Aug
  2. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks2 Apr
  3. 2024PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-Rails24 Feb
  4. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan
  5. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  6. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  7. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts7 Jun

Other references · 11

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  3. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  4. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  5. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  6. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
  7. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 208
  8. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  9. 2023Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competitioncited by 65
  10. 2018Unrestricted Adversarial Examplescited by 40
  11. 2024Failures to Find Transferable Image Jailbreaks Between Vision-Language Modelscited by 27

Built on by

Cited by year

Citing papers per year: 2025: 6.

6 papers.

  1. 2025STACK: Adversarial Attacks on LLM Safeguard Pipelinescited by 14
  2. 2025Chain-of-Thought Hijackingcited by 13
  3. 2025MIP against Agent: Malicious Image Patches Hijacking Multimodal OS Agentscited by 12
  4. 2025No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanismscited by 5
  5. 2025FLAME: Flexible LLM-Assisted Moderation Enginecited by 0
  6. 2025`A la recherche du sens perdu: your favourite LLM might have more to say than you can understandcited by 0