mlsanityData to 14 Oct 2026

A Security Risk Taxonomy for Prompt-Based Interaction With Large Language Models

arXiv 2311.11415 ↗First posted 19 Nov 2023cs.CR cs.AI cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

As large language models (LLMs) permeate more and more applications, an assessment of their associated security risks becomes increasingly necessary. The potential for exploitation by malicious actors, ranging from disinformation to data breaches and reputation damage, is substantial. This paper addresses a gap in current research by specifically focusing on security risks posed by LLMs within the prompt-based interaction scheme, which extends beyond the widely covered ethical and societal implications. Our work proposes a taxonomy of security risks along the user-model communication pipeline and categorizes the attacks by target and attack type alongside the commonly used confidentiality, integrity, and availability (CIA) triad. The taxonomy is reinforced with specific attack examples to showcase the real-world impact of these risks. Through this taxonomy, we aim to inform the development of robust and secure LLM applications, enhancing their safety and trustworthiness.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 42 of its 75 references that are in the map. First posted on arXiv 19 Nov 2023.

Builds on

42 of 75 references.

References · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  5. 2020Extracting Training Data from Large Language Modelscited by 2,189
  6. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  7. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 328
  8. 2021Membership Inference Attacks on Machine Learning: A Surveycited by 321
  9. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 296
  10. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
  11. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 195
  12. 2021Unlearnable Examples: Making Personal Data Unexploitablecited by 193
30 more
  1. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attackscited by 175
  2. 2021Understanding the Capabilities, Limitations, and Societal Impact of Large Language Modelscited by 160
  3. 2021Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Modelscited by 141
  4. 2021Adversarial Examples Make Strong Poisonscited by 119
  5. 2021Detoxifying Language Models Risks Marginalizing Minority Voicescited by 107
  6. 2023Regulating ChatGPT and other Large Generative AI Modelscited by 104
  7. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 93
  8. 2022Lost at C: A User Study on the Security Implications of Large Language Model Code Assistantscited by 91
  9. 2023Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicitycited by 89
  10. 2023Safety Assessment of Chinese Large Language Modelscited by 85
  11. 2023BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPTcited by 76
  12. 2019Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?cited by 71
  13. 2019Learning to Confuse: Generating Training Time Adversarial Data with Auto-Encodercited by 70
  14. 2022Theories of "Gender" in NLP Bias Researchcited by 62
  15. 2023LLM Platform Security: Applying a Systematic Evaluation Framework to OpenAI's ChatGPT Pluginscited by 51
  16. 2023Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreakscited by 49
  17. 2022Synthetic Disinformation Attacks on Automated Fact Verification Systemscited by 44
  18. 2021Mitigating harm in language models with conditional-likelihood filtrationcited by 40
  19. 2023From Text to MITRE Techniques: Exploring the Malicious Use of Large Language Models for Generating Cyber Attack Payloadscited by 32
  20. 2023MathAttack: Attacking Large Language Models Towards Math Solving Abilitycited by 32
  21. 2023Assessing Language Model Deployment with Risk Cardscited by 30
  22. 2021Generating Fake Cyber Threat Intelligence Using Transformer-Based Modelscited by 29
  23. 2023Quantifying Association Capabilities of Large Language Models and Its Implications on Privacy Leakagecited by 28
  24. 2023MoPe: Model Perturbation-based Privacy Attacks on Language Modelscited by 25
  25. 2022Targeted Phishing Campaigns using Large Scale Language Modelscited by 23
  26. 2022Adaptive Sampling Strategies to Construct Equitable Training Datasetscited by 18
  27. 2023LLMs Killed the Script Kiddie: How Agents Supported by Large Language Models Change the Landscape of Network Threat Testingcited by 17
  28. 2023Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvementscited by 14
  29. 2023LimeAttack: Local Explainable Method for Textual Hard-Label Adversarial Attackcited by 9
  30. 2023MeaeQ: Mount Model Extraction Attacks with Efficient Queriescited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 2; 2025: 7; 2026: 1.

12 papers.

  1. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  2. 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplificationcited by 81
  3. 2025Optimizing Retrieval Strategies for Financial Question Answering Documents in Retrieval-Augmented Generation Systemscited by 9
  4. 2025Breaking Agent Backbones: Evaluating the Security of Backbone LLMs in AI Agentscited by 8
  5. 2023Survey on AI Ethics: A Socio-technical Perspectivecited by 4
  6. 2025A Call to Action for a Secure-by-Design Generative AI Paradigmcited by 3
  7. 2025Cloud Investigation Automation Framework (CIAF): An AI-Driven Approach to Cloud Forensicscited by 3
  8. 2025Towards Safer Chatbots: Automated Policy Compliance Evaluation of Custom GPTscited by 1
  9. 2026Automating Cloud Security and Forensics Through a Secure-by-Design Generative AI Frameworkcited by 1
  10. 2024A Simple Architecture for Enterprise Large Language Model Applications based on Role based security and Clearance Levels using Retrieval-Augmented Generation or Mixture of Expertscited by 0
  11. 2025Analysing Safety Risks in LLMs Fine-Tuned with Pseudo-Malicious Cyber Security Datacited by 0
  12. 2025Quantifying Return on Security Controls in LLM Systemscited by 0