mlsanityData to 7 Oct 2026

Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks

arXiv 2302.05733 ↗First posted 11 Feb 2023cs.CR cs.LGcited by 268

Not on the map. Why?

Cited by 171 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent advances in instruction-following large language models (LLMs) have led to dramatic improvements in a range of NLP tasks. Unfortunately, we find that the same improved capabilities amplify the dual-use risks for malicious purposes of these models. Dual-use is difficult to prevent as instruction-following capabilities now enable standard attacks from computer security. The capabilities of these instruction-following LLMs provide strong economic incentives for dual-use by malicious actors. In particular, we show that instruction-following LLMs can produce targeted malicious content, including hate speech and scams, bypassing in-the-wild defenses implemented by LLM API vendors. Our analysis shows that this content can be generated economically and at cost likely lower than with human effort alone. Together, our findings suggest that LLMs will increasingly attract more sophisticated adversaries and attacks, and addressing these attacks may require new approaches to mitigations.

Key facts. Cited by 268 papers in the map (data through 7 Oct 2026). Builds on 28 of its 51 references that are in the map. First posted on arXiv 11 Feb 2023.

Builds on

28 of 51 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  8. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  9. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  10. 2021Ethical and social risks of harm from Language Modelscited by 957
  11. 2021Measuring Coding Challenge Competence With APPScited by 879
  12. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
16 more
  1. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  2. 2019Defending Against Neural Fake Newscited by 553
  3. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  4. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  5. 2021A Survey on Automated Fact-Checkingcited by 376
  6. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  7. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  8. 2022Predictability and Surprise in Large Generative Modelscited by 209
  9. 2020Toxicity Detection: Does Context Really Matter?cited by 93
  10. 2020The Radicalization Risks of GPT-3 and Advanced Neural Language Modelscited by 76
  11. 2020Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERTcited by 69
  12. 2020Robust Encodings: A Framework for Combating Adversarial Typoscited by 58
  13. 2021LIME: Learning Inductive Bias for Primitives of Mathematical Reasoningcited by 46
  14. 2022TempLM: Distilling Language Models into Template-Based Generatorscited by 4
  15. 2019A First Look at the Crypto-Mining Malware Ecosystem: A Decade of Unrestricted Wealthcited by 2
  16. 2017Economic Analysis of Ransomwarecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 48; 2024: 106; 2025: 77; 2026: 37.

268 papers.

Sort:

By area:

  1. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  2. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  3. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  4. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  5. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  6. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  7. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 476
  8. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  9. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  10. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 401
  11. 2024A StrongREJECT for Empty Jailbreakscited by 352
  12. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
38 more
  1. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  2. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  3. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 242
  4. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 237
  5. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  6. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 197
  7. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  8. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  9. 2023Can LLM-Generated Misinformation Be Detected?cited by 173
  10. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLMcited by 173
  11. 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 165
  12. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  13. 2023In-Context Impersonation Reveals Large Language Models' Strengths and Biasescited by 154
  14. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 146
  15. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 129
  16. 2024Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systemscited by 126
  17. 2024LLM Agents can Autonomously Exploit One-day Vulnerabilitiescited by 125
  18. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  19. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performancecited by 111
  20. 2024LLM Agents can Autonomously Hack Websitescited by 108
  21. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Modelscited by 98
  22. 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Modelscited by 95
  23. 2024Teams of LLM Agents can Exploit Zero-Day Vulnerabilitiescited by 93
  24. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
  25. 2023Identifying and Mitigating the Security Risks of Generative AIcited by 90
  26. 2024Evolutionary Computation in the Era of Large Language Model: Survey and Roadmapcited by 89
  27. 2024Coercing LLMs to do and reveal (almost) anythingcited by 88
  28. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMscited by 87
  29. 2023Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinkingcited by 78
  30. 2024Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Modelscited by 77
  31. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 76
  32. 2023Attack Prompt Generation for Red Teaming and Defending Large Language Modelscited by 73
  33. 2024Safeguarding Large Language Models: A Surveycited by 73
  34. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 73
  35. 2024CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completioncited by 70
  36. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  37. 2025Design Patterns for Securing LLM Agents against Prompt Injectionscited by 67
  38. 2023Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competitioncited by 65

50 most cited of 268 All 268 (JSON)