mlsanityData to 7 Oct 2026

ASTPrompter: Preference-Aligned Automated Language Model Red-Teaming to Generate Low-Perplexity Unsafe Prompts

arXiv 2407.09447 ↗First posted 12 Jul 2024cs.CLcited by 3

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Existing LLM red-teaming approaches prioritize high attack success rate, often resulting in high-perplexity prompts. This focus overlooks low-perplexity attacks that are more difficult to filter, more likely to arise during benign usage, and more impactful as negative downstream training examples. In response, we introduce ASTPrompter, a single-step optimization method that uses contrastive preference learning to train an attacker to maintain low perplexity while achieving a high attack success rate (ASR). ASTPrompter achieves an attack success rate 5.1 times higher on Llama-8.1B while using inputs that are 2.1 times more likely to occur according to the frozen LLM. Furthermore, our attack transfers to Mistral-7B, Qwen-7B, and TinyLlama in both black- and white-box settings. Lastly, by tuning a single hyperparameter in our method, we discover successful attack prefixes along an efficient frontier between ASR and perplexity, highlighting perplexity as a previously under-considered factor in red-teaming.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 60 references that are in the map. First posted on arXiv 12 Jul 2024.

Builds on

30 of 60 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models1 Sep
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 28

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2023Qwen Technical Reportcited by 3,273
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  4. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  5. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  6. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  7. 2022Red Teaming Language Models with Language Modelscited by 983
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  9. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  10. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  11. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  12. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
16 more
  1. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  2. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  3. 2023Pretraining Language Models with Human Preferencescited by 245
  4. 2021Latent Hatred: A Benchmark for Understanding Implicit Hate Speechcited by 216
  5. 2019Racial Bias in Hate Speech and Abusive Language Detection Datasetscited by 208
  6. 2022Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Datasetcited by 113
  7. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 111
  8. 2022Controllable Natural Language Generation with Contrastive Prefixescited by 75
  9. 2020ConvoKit: A Toolkit for the Analysis of Conversationscited by 71
  10. 2019Adaptive Stress Testing for Autonomous Vehiclescited by 70
  11. 2024Gradient-Based Language Model Red Teamingcited by 33
  12. 2022Why So Toxic? Measuring and Triggering Toxic Behavior in Open-Domain Chatbotscited by 31
  13. 2021Automatic Evaluation and Moderation of Open-domain Dialogue Systemscited by 28
  14. 2024"They are uncultured": Unveiling Covert Harms and Social Threats in LLM Generated Conversationscited by 24
  15. 2023Automated Ableism: An Exploration of Explicit Disability Biases in Sentiment and Toxicity Analysis Modelscited by 20
  16. 2024FamiCom: Further Demystifying Prompts for Language Models with Task-Agnostic Performance Estimationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Characterizing the Robustness of Black-Box LLM Planners Under Perturbed Observations with Adaptive Stress Testingcited by 1
  2. 2025Aircraft Collision Avoidance Systems: Technological Challenges and Solutions on the Path to Regulatory Acceptancecited by 0
  3. 2026SecureForge: Finding and Preventing Vulnerabilities in LLM-Generated Code via Prompt Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Code-Switching Red-Teaming: LLM Evaluation for Safety and Multilingual Understanding
  2. 2024Against The Achilles' Heel: A Survey on Red Teaming for Generative Models
  3. 2026AutoRISE: Agent-Driven Strategy Evolution for Red-Teaming Large Language Models
  4. 2026AMT-X: Phase-Structured Multi-Turn Red-Teaming with Checklist-Gated Evaluation
  5. 2025A methodological analysis of prompt perturbations and their effect on attack success rates
  6. 2026FlashRT: Towards Computationally and Memory Efficient Red-Teaming for Prompt Injection and Knowledge Corruption
  7. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teaming
  8. 2025AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agents