mlsanityData to 7 Oct 2026

PAL: Proxy-Guided Black-Box Attack on Large Language Models

arXiv 2402.09674 ↗First posted 15 Feb 2024cs.CL cs.AI cs.CRcited by 46

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Large Language Models (LLMs) have surged in popularity in recent months, but they have demonstrated concerning capabilities to generate harmful content when manipulated. While techniques like safety fine-tuning aim to minimize harmful use, recent works have shown that LLMs remain vulnerable to attacks that elicit toxic responses. In this work, we introduce the Proxy-Guided Attack on LLMs (PAL), the first optimization-based attack on LLMs in a black-box query-only setting. In particular, it relies on a surrogate model to guide the optimization and a sophisticated loss designed for real-world LLM APIs. Our attack achieves 84% attack success rate (ASR) on GPT-3.5-Turbo and 48% on Llama-2-7B, compared to 4% for the current state of the art. We also propose GCG++, an improvement to the GCG attack that reaches 94% ASR on white-box Llama-2-7B, and the Random-Search Attack on LLMs (RAL), a strong but simple baseline for query-based attacks. We believe the techniques proposed in this work will enable more comprehensive safety testing of LLMs and, in the long term, the development of better security guardrails. The code can be found at https://github.com/chawins/pal.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 43 of its 105 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

43 of 105 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb
  2. 2022Red Teaming Language Models with Language Models7 Feb

Other references · 41

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  6. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  10. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  11. 2021Ethical and social risks of harm from Language Modelscited by 957
  12. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
29 more
  1. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  2. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  3. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  4. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  5. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  6. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  7. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  8. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  9. 2019On Evaluating Adversarial Robustnesscited by 442
  10. 2023Low-Resource Languages Jailbreak GPT-4cited by 290
  11. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  12. 2023Pretraining Language Models with Human Preferencescited by 245
  13. 2018Adversarial Risk and the Dangers of Evaluating Against Weak Attackscited by 221
  14. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  15. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Modelscited by 128
  16. 2019Improving Black-box Adversarial Attacks with a Transfer-based Priorcited by 121
  17. 2023Language Model Inversioncited by 66
  18. 2022Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examplescited by 65
  19. 2019Black-Box Adversarial Attack with Transferable Model-based Embeddingcited by 57
  20. 2019Stateful Detection of Black-Box Adversarial Attackscited by 55
  21. 2021Model Extraction and Adversarial Transferability, Your BERT is Vulnerable!cited by 51
  22. 2019Subspace Attack: Exploiting Promising Subspaces for Query-Efficient Black-box Attackscited by 35
  23. 2024All in How You Ask for It: Simple Black-Box Method for Jailbreak Attackscited by 30
  24. 2023LoFT: Local Proxy Fine-tuning For Improving Transferability Of Adversarial Attacks Against Large Language Modelcited by 21
  25. 2022Attacking deep networks with surrogate-based adversarial black-box methods is easycited by 18
  26. 2022Blackbox Attacks via Surrogate Ensemble Searchcited by 13
  27. 2023Improving Adversarial Transferability via Model Alignmentcited by 10
  28. 2023PubDef: Defending Against Transfer Attacks From Public Modelscited by 7
  29. 2022Publishing Efficient On-device Models Increases Adversarial Vulnerabilitycited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 22; 2025: 17; 2026: 7.

46 papers.

Sort:

By area:

  1. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  2. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  3. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  4. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 145
  5. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  6. 2024Does Refusal Training in LLMs Generalize to the Past Tense?cited by 74
  7. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 68
  8. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  9. 2024SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Mannercited by 57
  10. 2024Query-Based Adversarial Prompt Generationcited by 55
  11. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignmentcited by 42
  12. 2025Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safetycited by 38
34 more
  1. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 28
  2. 2024Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learningcited by 23
  3. 2025Understanding and Enhancing the Transferability of Jailbreaking Attackscited by 22
  4. 2024Improved Generation of Adversarial Examples Against Safety-aligned LLMscited by 17
  5. 2025SoK: Evaluating Jailbreak Guardrails for Large Language Modelscited by 17
  6. 2024Does Safety Training of LLMs Generalize to Semantically Related Natural Prompts?cited by 15
  7. 2025Fun-tuning: Characterizing the Vulnerability of Proprietary LLMs to Optimization-based Prompt Injection Attacks via the Fine-Tuning Interfacecited by 15
  8. 2024Jailbreaking as a Reward Misspecification Problemcited by 11
  9. 2024LLMs Can Defend Themselves Against Jailbreaking in a Practical Manner: A Vision Papercited by 9
  10. 2025GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methodscited by 9
  11. 2025DualBreach: Efficient Dual-Jailbreaking via Target-Driven Initialization and Multi-Target Optimizationcited by 9
  12. 2024Autonomous LLM-Enhanced Adversarial Attack for Text-to-Motioncited by 7
  13. 2025Black-box Optimization of LLM Outputs by Asking for Directionscited by 7
  14. 2024Deciphering the Chaos: Enhancing Jailbreak Attacks via Adversarial Prompt Translationcited by 6
  15. 2024Adversarial Vulnerabilities in Large Language Models for Time Series Forecastingcited by 6
  16. 2025ASTRA: Autonomous Spatial-Temporal Red-teaming for AI Software Assistantscited by 6
  17. 2025Chain-of-Lure: A Universal Jailbreak Attack Framework using Unconstrained Synthetic Narrativescited by 5
  18. 2024Investigating Adversarial Trigger Transfer in Large Language Modelscited by 4
  19. 2025KDA: A Knowledge-Distilled Attacker for Generating Diverse Prompts to Jailbreak LLMscited by 4
  20. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 4
  21. 2024LIAR: Leveraging Inference Time Alignment (Best-of-N) to Jailbreak LLMs in Secondscited by 3
  22. 2026Safety Recovery in Reasoning Models Is Only a Few Early Steering Steps Awaycited by 3
  23. 2026Stop Tracking Me! Proactive Defense Against Attribute Inference Attack in LLMscited by 3
  24. 2026OTora: A Unified Red Teaming Framework for Reasoning-Level Denial-of-Service in LLM Agentscited by 3
  25. 2024On the Validity of Traditional Vulnerability Scoring Systems for Adversarial Attacks against LLMscited by 2
  26. 2025Checkpoint-GCG: Auditing and Attacking Fine-Tuning-Based Prompt Injection Defensescited by 2
  27. 2026SoK: Robustness in Large Language Models against Jailbreak Attackscited by 2
  28. 2025SoK: A Comprehensive Security Analysis of Jailbreak Resilience in GPT and DeepSeek Modelscited by 1
  29. 2025Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generationcited by 1
  30. 2026Security in the Fine-Tuning Lifecycle of Large Language Models: Threats, Defenses,Evaluation, and Future Directionscited by 1
  31. 2025TestDG: Test-time Domain Generalization for Continual Test-time Adaptationcited by 0
  32. 2025Adversarial Attacks in Multimodal Systems: A Practitioner's Surveycited by 0
  33. 2026FlashRT: Towards Computationally and Memory Efficient Red-Teaming for Prompt Injection and Knowledge Corruptioncited by 0
  34. 2026Overloading Large Vision-Language Models for Jailbreakingcited by 0