mlsanityData to 14 Oct 2026

Distillability of LLM Security Logic: Predicting Attack Success Rate of Outline Filling Attack via Ranking Regression

arXiv 2511.22044 ↗First posted 27 Nov 2025cs.CR cs.AIcited by 0

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

In the realm of black-box jailbreak attacks on large language models (LLMs), the feasibility of constructing a narrow safety proxy, a lightweight model designed to predict the attack success rate (ASR) of adversarial prompts, remains underexplored. This work investigates the distillability of an LLM's core security logic. We propose a novel framework that incorporates an improved outline filling attack to achieve dense sampling of the model's security boundaries. Furthermore, we introduce a ranking regression paradigm that replaces standard regression and trains the proxy model to predict which prompt yields a higher ASR. Experimental results show that our proxy model achieves an accuracy of 91.1 percent in predicting the relative ranking of average long response (ALR), and 69.2 percent in predicting ASR. These findings confirm the predictability and distillability of jailbreak behaviors, and demonstrate the potential of leveraging such distillability to optimize black-box attacks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 13 of its 28 references that are in the map. First posted on arXiv 27 Nov 2025.

Builds on

13 of 28 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024GASP: Efficient Black-Box Generation of Adversarial Suffixes for Jailbreaking LLMs21 Nov
  2. 2024AttnGCG: Enhancing Jailbreaking Attacks on LLMs with Attention Manipulation11 Oct
  3. 2024You Know What I'm Saying: Jailbreak Attack via Implicit Reference4 Oct
  4. 2024Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characters30 May
  5. 2024DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakers25 Feb
  6. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  7. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 5

  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  2. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  3. 2024Does Refusal Training in LLMs Generalize to the Past Tense?cited by 79
  4. 2025RL Is a Hammer and LLMs Are Nails: A Simple Reinforcement Learning Recipe for Strong Prompt Injectioncited by 36
  5. 2025HarmMetric Eval: Benchmarking Metrics and Judges for LLM Harmfulness Assessmentcited by 15

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SoK: Robustness in Large Language Models against Jailbreak Attacks
  2. 2025Adversarial Prompt Evaluation: Systematic Benchmarking of Guardrails Against Prompt Input Attacks on LLMs
  3. 2026Quantifying LLM Safety Degradation Under Repeated Attacks Using Survival Analysis
  4. 2025Efficient and Stealthy Jailbreak Attacks via Adversarial Prompt Distillation from LLMs to SLMs
  5. 2026MultiBreak: A Scalable and Diverse Multi-turn Jailbreak Benchmark for Evaluating LLM Safety
  6. 2025SecurityLingua: Efficient Defense of LLM Jailbreak Attacks via Security-Aware Prompt Compression
  7. 2025Bypassing LLM Guardrails: An Empirical Analysis of Evasion Attacks against Prompt Injection and Jailbreak Detection Systems
  8. 2025Paper Summary Attack: Jailbreaking LLMs through LLM Safety Papers