mlsanityData to 14 Oct 2026

LLM Security: Vulnerabilities, Attacks, Defenses, and Countermeasures

arXiv 2505.01177 ↗First posted 2 May 2025cs.CR cs.AI cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

As large language models (LLMs) continue to evolve, it is critical to assess the security threats and vulnerabilities that may arise both during their training phase and after models have been deployed. This survey seeks to define and categorize the various attacks targeting LLMs, distinguishing between those that occur during the training phase and those that affect already trained models. A thorough analysis of these attacks is presented, alongside an exploration of defense mechanisms designed to mitigate such threats. Defenses are classified into two primary categories: prevention-based and detection-based defenses. Furthermore, our survey summarizes possible attacks and their corresponding defense strategies. It also provides an evaluation of the effectiveness of the known defense mechanisms for the different security threats. Our survey aims to offer a structured framework for securing LLMs, while also identifying areas that require further research to improve and strengthen defenses against emerging security challenges.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 81 of its 155 references that are in the map. First posted on arXiv 2 May 2025.

Builds on

81 of 155 references.

References · 81

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  6. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  7. 2013Intriguing properties of neural networkscited by 5,155
  8. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  9. 2020Extracting Training Data from Large Language Modelscited by 2,189
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  11. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  12. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
38 more
  1. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  2. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,106
  3. 2019Deep Leakage from Gradientscited by 1,021
  4. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  5. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 712
  6. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  7. 2016Stealing Machine Learning Models via Prediction APIscited by 619
  8. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  9. 2023Prompt Injection attack against LLM-integrated Applicationscited by 605
  10. 2023Detecting Language Model Attacks with Perplexitycited by 431
  11. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 411
  12. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 394
  13. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
  14. 2023Explainability for Large Language Models: A Surveycited by 333
  15. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  16. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 314
  17. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 290
  18. 2023Preference Ranking Optimization for Human Alignmentcited by 285
  19. 2017MagNet: a Two-Pronged Defense against Adversarial Examplescited by 269
  20. 2019The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networkscited by 246
  21. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 244
  22. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 234
  23. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 200
  24. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 194
  25. 2023Text Embeddings Reveal (Almost) As Much As Textcited by 192
  26. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 186
  27. 2021Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfercited by 158
  28. 2024Automatic and Universal Prompt Injection Attacks against Large Language Modelscited by 119
  29. 2021Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoningcited by 113
  30. 2021Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Modelscited by 105
  31. 2021RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Modelscited by 104
  32. 2024On Protecting the Data Privacy of Large Language Models (LLMs): A Surveycited by 96
  33. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMscited by 95
  34. 2023Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Modelscited by 94
  35. 2020Trojaning Language Models for Fun and Profitcited by 90
  36. 2021Hidden Backdoors in Human-Centric Language Modelscited by 90
  37. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 90
  38. 2024Fast Adversarial Attacks on Language Models In One GPU Minutecited by 88

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 3.

8 papers.

  1. 2025BadThink: Triggered Overthinking Attacks on Chain-of-Thought Reasoning in Large Language Modelscited by 8
  2. 2025Uncovering Vulnerabilities of LLM-Assisted Cyber Threat Intelligencecited by 6
  3. 2025Memorization in Large Language Models in Medicine: Prevalence, Characteristics, and Implicationscited by 5
  4. 2025SoK: Semantic Privacy in Large Language Modelscited by 3
  5. 2025Are All Prompt Components Value-Neutral? Understanding the Heterogeneous Adversarial Robustness of Dissected Prompt in Large Language Modelscited by 3
  6. 2026Architecture Matters for Multi-Agent Securitycited by 3
  7. 2026PRISM: Generation-Time Detection and Mitigation of Secret Leakage in Multi-Agent LLM Pipelinescited by 2
  8. 2026A Systematic Review of Algorithmic Red Teaming Methodologies for Assurance and Security of AI Applicationscited by 1