mlsanityData to 14 Oct 2026

An LLM can Fool Itself: A Prompt-Based Adversarial Attack

arXiv 2310.13345 ↗First posted 20 Oct 2023cs.CRcited by 116

Not on the map. Why?

Cited by 31 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The wide-ranging applications of large language models (LLMs), especially in safety-critical domains, necessitate the proper evaluation of the LLM's adversarial robustness. This paper proposes an efficient tool to audit the LLM's adversarial robustness via a prompt-based adversarial attack (PromptAttack). PromptAttack converts adversarial textual attacks into an attack prompt that can cause the victim LLM to output the adversarial sample to fool itself. The attack prompt is composed of three important components: (1) original input (OI) including the original sample and its ground-truth label, (2) attack objective (AO) illustrating a task description of generating a new sample that can fool itself without changing the semantic meaning, and (3) attack guidance (AG) containing the perturbation instructions to guide the LLM on how to complete the task by perturbing the original sample at character, word, and sentence levels, respectively. Besides, we use a fidelity filter to ensure that PromptAttack maintains the original semantic meanings of the adversarial examples. Further, we enhance the attack power of PromptAttack by ensembling adversarial examples at different perturbation levels. Comprehensive empirical results using Llama2 and GPT-3.5 validate that PromptAttack consistently yields a much higher attack success rate compared to AdvGLUE and AdvGLUE++. Interesting findings include that a simple emoji can easily mislead GPT-3.5 to make wrong predictions.

Key facts. Cited by 116 papers in the map (data through 14 Oct 2026). First posted on arXiv 20 Oct 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 45; 2025: 46; 2026: 24.

116 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  2. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  3. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 80
  4. 2024Defending LLMs against Jailbreaking Attacks via Backtranslationcited by 70
  5. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 52
  6. 2025VeriGuard: Enhancing LLM Agent Safety via Verified Code Generationcited by 49
  7. 2024Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Modelscited by 44
  8. 2024Multi-hop Question Answering under Temporal Knowledge Editingcited by 36
  9. 2025Fraud-R1 : A Multi-Round Benchmark for Assessing the Robustness of LLM Against Augmented Fraud and Phishing Inducementscited by 32
  10. 2024Large Model Based Agents: State-of-the-Art, Cooperation Paradigms, Security and Privacy, and Future Trendscited by 26
  11. 2025CheatAgent: Attacking LLM-Empowered Recommender Systems via LLM Agentcited by 25
  12. 2024Editable Concept Bottleneck Modelscited by 23
38 more
  1. 2024Transferable Adversarial Attacks on SAM and Its Downstream Modelscited by 21
  2. 2024LLM Task Interference: An Initial Study on the Impact of Task-Switch in Conversational Historycited by 19
  3. 2024Prompt-SAW: Leveraging Relation-Aware Graphs for Textual Prompt Compressioncited by 18
  4. 2025CPA-RAG:Covert Poisoning Attacks on Retrieval-Augmented Generation in Large Language Modelscited by 17
  5. 2025UniGuardian: A Unified Defense for Detecting Prompt Injection, Backdoor Attacks and Adversarial Attacks in Large Language Modelscited by 16
  6. 2024Attacks on Third-Party APIs of Large Language Modelscited by 15
  7. 2024LLM Whisperer: An Inconspicuous Attack to Bias LLM Responsescited by 15
  8. 2024Do LLMs dream of elephants (when told not to)? Latent concept association and associative memory in transformerscited by 15
  9. 2024Causal-Guided Active Learning for Debiasing Large Language Modelscited by 15
  10. 2024Securing Vision-Language Models with a Robust Encoder Against Jailbreak and Adversarial Attackscited by 15
  11. 2025Is Your LLM-Based Multi-Agent a Reliable Real-World Planner? Exploring Fraud Detection in Travel Planningcited by 15
  12. 2024Cutting Through the Noise: Boosting LLM Performance on Math Word Problemscited by 14
  13. 2024PertEval: Unveiling Real Knowledge Capacity of LLMs with Knowledge-Invariant Perturbationscited by 13
  14. 2025Standard Benchmarks Fail -- Auditing LLM Agents in Finance Must Prioritize Riskcited by 13
  15. 2024Leveraging Logical Rules in Knowledge Editing: A Cherry on the Topcited by 12
  16. 2024GASP: Efficient Black-Box Generation of Adversarial Suffixes for Jailbreaking LLMscited by 12
  17. 2024Dialectical Alignment: Resolving the Tension of 3H and Security Threats of LLMscited by 11
  18. 2024Large Language Model Sentinel: LLM Agent for Adversarial Purificationcited by 10
  19. 2024CheckEmbed: Effective Verification of LLM Solutions to Open-Ended Taskscited by 10
  20. 2024Attacking Misinformation Detection Using Adversarial Examples Generated by Language Modelscited by 10
  21. 2024SelfPrompt: Autonomously Evaluating LLM Robustness via Domain-Constrained Knowledge Guidelines and Refined Adversarial Promptscited by 10
  22. 2025Single-pass Detection of Jailbreaking Input in Large Language Modelscited by 10
  23. 2025Response Attack: Exploiting Contextual Priming to Jailbreak Large Language Modelscited by 10
  24. 2024MQA-KEAL: Multi-hop Question Answering under Knowledge Editing for Arabic Languagecited by 9
  25. 2025TAI3: Testing Agent Integrity in Interpreting User Intentcited by 9
  26. 2025From Pretrain to Pain: Adversarial Vulnerability of Video Foundation Models Without Task Knowledgecited by 9
  27. 2024Testing the Limits of Jailbreaking Defenses with the Purple Problemcited by 8
  28. 2024Multitask Mayhem: Unveiling and Mitigating Safety Gaps in LLMs Fine-tuningcited by 8
  29. 2024Recent advancements in LLM Red-Teaming: Techniques, Defenses, and Ethical Considerationscited by 8
  30. 2024Stepwise Reasoning Error Disruption Attack of LLMscited by 8
  31. 2025TH-Bench: Evaluating Evading Attacks via Humanizing AI Text on Machine-Generated Text Detectorscited by 8
  32. 2025FLEX: A Benchmark for Evaluating Robustness of Fairness in Large Language Modelscited by 8
  33. 2026Disentangling Adversarial Prompts: A Semantic-Graph Defense for Robust LLM Securitycited by 8
  34. 2024MONAL: Model Autophagy Analysis for Modeling Human-AI Interactionscited by 7
  35. 2024An Empirical Categorization of Prompting Techniques for Large Language Models: A Practitioner's Guidecited by 7
  36. 2025The TIP of the Iceberg: Revealing a Hidden Class of Task-in-Prompt Adversarial Attacks on LLMscited by 7
  37. 2025J&H: Evaluating the Robustness of Large Language Models Under Knowledge-Injection Attacks in Legal Domaincited by 7
  38. 2025Retrieval-Augmented Purifier for Robust LLM-Empowered Recommendationcited by 7

50 most cited of 116 All 116 (JSON)