# Exploring Vulnerabilities and Protections in Large Language Models: A Survey

arXiv: https://arxiv.org/abs/2406.00240 · HTML: https://mlsanity.com/paper/2406.00240/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 31 of its 42 references that are in the map.
- First posted on arXiv 1 Jun 2024.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 30 of 31)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [Baseline Defenses for Adversarial Attacks Against Aligned Language Models](https://mlsanity.com/paper/2309.00614/) (2023)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study](https://mlsanity.com/paper/2305.13860/) (2023)
- [SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks](https://mlsanity.com/paper/2310.03684/) (2023)
- [A Holistic Approach to Undesired Content Detection in the Real World](https://mlsanity.com/paper/2208.03274/) (2022)
- [Poisoning Language Models During Instruction Tuning](https://mlsanity.com/paper/2305.00944/) (2023)
- [ONION: A Simple and Effective Defense Against Textual Backdoor Attacks](https://mlsanity.com/paper/2011.10369/) (2020)
- [Demystifying Prompts in Language Models via Perplexity Estimation](https://mlsanity.com/paper/2212.04037/) (2022)
- [Security and Privacy Challenges of Large Language Models: A Survey](https://mlsanity.com/paper/2402.00888/) (2024)
- [Open Sesame! Universal Black Box Jailbreaking of Large Language Models](https://mlsanity.com/paper/2309.01446/) (2023)
- [ProPILE: Probing Privacy Leakage in Large Language Models](https://mlsanity.com/paper/2307.01881/) (2023)
- [Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoning](https://mlsanity.com/paper/2108.13888/) (2021)
- [RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Models](https://mlsanity.com/paper/2110.07831/) (2021)
- [Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Models](https://mlsanity.com/paper/2305.01219/) (2023)
- [Trojaning Language Models for Fun and Profit](https://mlsanity.com/paper/2008.00312/) (2020)
- [Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods](https://mlsanity.com/paper/2210.07321/) (2022)
- [A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks](https://mlsanity.com/paper/2206.08514/) (2022)
- [Composite Backdoor Attacks Against Large Language Models](https://mlsanity.com/paper/2310.07676/) (2023)
- [Adversarial Demonstration Attacks on Large Language Models](https://mlsanity.com/paper/2305.14950/) (2023)
- [TrojanPuzzle: Covertly Poisoning Code-Suggestion Models](https://mlsanity.com/paper/2301.02344/) (2023)
- [Fine-mixing: Mitigating Backdoors in Fine-tuned Language Models](https://mlsanity.com/paper/2210.09545/) (2022)
- [A Comprehensive Overview of Backdoor Attacks in Large Language Models within Communication Networks](https://mlsanity.com/paper/2308.14367/) (2023)
- [Prompt Packer: Deceiving LLMs through Compositional Instruction with Hidden Attacks](https://mlsanity.com/paper/2310.10077/) (2023)
- [Backdoor Attacks and Defenses in Federated Learning: Survey, Challenges and Future Research Directions](https://mlsanity.com/paper/2303.02213/) (2023)
- [Defending Pre-trained Language Models as Few-shot Learners against Backdoor Attacks](https://mlsanity.com/paper/2309.13256/) (2023)

## Built on by (most cited first, 11 of 11)

- [The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs](https://mlsanity.com/paper/2506.11094/) (2025)
- [When Models Reason in Your Language: Controlling Thinking Language Comes at the Cost of Accuracy](https://mlsanity.com/paper/2505.22888/) (2025)
- [LLM Security: Vulnerabilities, Attacks, Defenses, and Countermeasures](https://mlsanity.com/paper/2505.01177/) (2025)
- [Adversarial Attacks on LLM-as-a-Judge Systems: Insights from Prompt Injections](https://mlsanity.com/paper/2504.18333/) (2025)
- [POISONCRAFT: Practical Poisoning of Retrieval-Augmented Generation for Large Language Models](https://mlsanity.com/paper/2505.06579/) (2025)
- [eFedLLM: Efficient LLM Inference Based on Federated Learning](https://mlsanity.com/paper/2411.16003/) (2024)
- [On the Validity of Traditional Vulnerability Scoring Systems for Adversarial Attacks against LLMs](https://mlsanity.com/paper/2412.20087/) (2024)
- [Layered LLM Defenses as an Ensemble: Access Tiers, Inference Cost, and the Measured Failure Correlation Between Defense Layers](https://mlsanity.com/paper/2608.28327/) (2026)
- [Hallucinating AI Hijacking Attack: Large Language Models and Malicious Code Recommenders](https://mlsanity.com/paper/2410.06462/) (2024)
- [Bridging Today and the Future of Humanity: AI Safety in 2024 and Beyond](https://mlsanity.com/paper/2410.18114/) (2024)
- [Enhancing Social Robots through Resilient AI](https://mlsanity.com/paper/2510.21469/) (2025)

## Abstract (arXiv)

As Large Language Models (LLMs) increasingly become key components in various AI applications, understanding their security vulnerabilities and the effectiveness of defense mechanisms is crucial. This survey examines the security challenges of LLMs, focusing on two main areas: Prompt Hacking and Adversarial Attacks, each with specific types of threats. Under Prompt Hacking, we explore Prompt Injection and Jailbreaking Attacks, discussing how they work, their potential impacts, and ways to mitigate them. Similarly, we analyze Adversarial Attacks, breaking them down into Data Poisoning Attacks and Backdoor Attacks. This structured examination helps us understand the relationships between these vulnerabilities and the defense strategies that can be implemented. The survey highlights these security challenges and discusses robust defensive frameworks to protect LLMs against these threats. By detailing these security issues, the survey contributes to the broader discussion on creating resilient AI systems that can resist sophisticated attacks.