mlsanityData to 14 Oct 2026

KDA: A Knowledge-Distilled Attacker for Generating Diverse Prompts to Jailbreak LLMs

arXiv 2502.05223 ↗First posted 5 Feb 2025cs.CR cs.AI cs.CLcited by 5

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Jailbreak attacks exploit specific prompts to bypass LLM safeguards, causing the LLM to generate harmful, inappropriate, and misaligned content. Current jailbreaking methods rely heavily on carefully designed system prompts and numerous queries to achieve a single successful attack, which is costly and impractical for large-scale red-teaming. To address this challenge, we propose to distill the knowledge of an ensemble of SOTA attackers into a single open-source model, called Knowledge-Distilled Attacker (KDA), which is finetuned to automatically generate coherent and diverse attack prompts without the need for meticulous system prompt engineering. Compared to existing attackers, KDA achieves higher attack success rates and greater cost-time efficiency when targeting multiple SOTA open-source and commercial black-box LLMs. Furthermore, we conducted a quantitative diversity analysis of prompts generated by baseline methods and KDA, identifying diverse and ensemble attacks as key factors behind KDA's effectiveness and efficiency.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 31 of its 38 references that are in the map. First posted on arXiv 5 Feb 2025.

Builds on

31 of 38 references.

Closest ancestors · 21

Same direction, newest first

  1. 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs3 Oct
  2. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs21 Apr
  3. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs11 Apr
  4. 2024Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstruction28 Feb
  5. 2024CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Models26 Feb
  6. 2024ASETF: A Novel Method for Jailbreak Attack on LLMs through Translate Suffix Embeddings25 Feb
  7. 2024Semantic Mirror Jailbreak: Genetic Algorithm Based Jailbreak Prompts Against Open-source LLMs21 Feb
  8. 2024Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Clues14 Feb
  9. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability13 Feb
  10. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan
  11. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  12. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily14 Nov
9 more
  1. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreaker6 Nov
  2. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  3. 2023Low-Resource Languages Jailbreak GPT-43 Oct
  4. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  5. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Models4 Sep
  6. 2023Detecting Language Model Attacks with Perplexity27 Aug
  7. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models7 Aug
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  9. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 10

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  3. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  4. 2023BloombergGPT: A Large Language Model for Financecited by 939
  5. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 333
  6. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 216
  7. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 205
  8. 2024Detoxifying Large Language Models via Knowledge Editingcited by 87
  9. 2021Fine-Tuning Large Neural Language Models for Biomedical Natural Language Processingcited by 57
  10. 2024PAL: Proxy-Guided Black-Box Attack on Large Language Modelscited by 50

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025DiffuGuard: How Intrinsic Safety is Lost and Found in Diffusion Large Language Modelscited by 12
  2. 2025SECA: Semantically Equivalent and Coherent Attacks for Eliciting LLM Hallucinationscited by 9
  3. 2025Response-Based Knowledge Distillation for Multilingual Jailbreak Prevention Unwittingly Compromises Safetycited by 4
  4. 2026REALISTA: Realistic Latent Adversarial Attacks that Elicit LLM Hallucinationscited by 1
  5. 2026REPRO: Reflective Evolutionary Prompt Optimizer for Automated LLM Red-Teamingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Knowledge-to-Jailbreak: Investigating Knowledge-driven Jailbreaking Attacks for Large Language Models
  2. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defenses
  3. 2026The Art of the Jailbreak: Formulating Jailbreak Attacks for LLM Security Beyond Binary Scoring
  4. 2026Re-Triggering Safeguards within LLMs for Jailbreak Detection
  5. 2024SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Manner
  6. 2025Beyond Fixed and Dynamic Prompts: Embedded Jailbreak Templates for Advancing LLM Security
  7. 2024Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attacks
  8. 2026ATTCK-R: Knowledge-Aware Long-Horizon Jailbreaking for Protected LLMs