# Security Concerns for Large Language Models: A Survey

arXiv: https://arxiv.org/abs/2505.18889 · HTML: https://mlsanity.com/paper/2505.18889/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 63 of its 92 references that are in the map.
- First posted on arXiv 24 May 2025.

## Builds on (most cited first, 30 of 63)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023)
- [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022)
- [A Watermark for Large Language Models](https://mlsanity.com/paper/2301.10226/) (2023)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [Measuring Faithfulness in Chain-of-Thought Reasoning](https://mlsanity.com/paper/2307.13702/) (2023)
- [Tree of Attacks: Jailbreaking Black-Box LLMs Automatically](https://mlsanity.com/paper/2312.02119/) (2023)
- [Safe RLHF: Safe Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2310.12773/) (2023)
- [Baseline Defenses for Adversarial Attacks Against Aligned Language Models](https://mlsanity.com/paper/2309.00614/) (2023)
- [Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training](https://mlsanity.com/paper/2401.05566/) (2024)
- [The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions](https://mlsanity.com/paper/2404.13208/) (2024)
- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation](https://mlsanity.com/paper/2503.11926/) (2025)
- [Formalizing and Benchmarking Prompt Injection Attacks and Defenses](https://mlsanity.com/paper/2310.12815/) (2023)
- [Deliberative Alignment: Reasoning Enables Safer Language Models](https://mlsanity.com/paper/2412.16339/) (2024)
- [Frontier Models are Capable of In-context Scheming](https://mlsanity.com/paper/2412.04984/) (2024)
- [Poisoning Language Models During Instruction Tuning](https://mlsanity.com/paper/2305.00944/) (2023)
- [Rethinking Machine Unlearning for Large Language Models](https://mlsanity.com/paper/2402.08787/) (2024)
- [Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment](https://mlsanity.com/paper/2308.09662/) (2023)
- [AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents](https://mlsanity.com/paper/2503.18666/) (2025)
- [Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks](https://mlsanity.com/paper/2310.10844/) (2023)
- [Security and Privacy Challenges of Large Language Models: A Survey](https://mlsanity.com/paper/2402.00888/) (2024)
- [Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection](https://mlsanity.com/paper/2307.16888/) (2023)

## Built on by (most cited first, 29 of 29)

- [Enforcing Temporal Constraints for LLM Agents](https://mlsanity.com/paper/2512.23738/) (2025)
- [Security Threat Modeling for Emerging AI-Agent Protocols: A Comparative Analysis of MCP, A2A, Agora, and ANP](https://mlsanity.com/paper/2602.11327/) (2026)
- [KV Cache Transform Coding for Compact Storage in LLM Inference](https://mlsanity.com/paper/2511.01815/) (2025)
- [A Benchmark for Evaluating Outcome-Driven Constraint Violations in Autonomous AI Agents](https://mlsanity.com/paper/2512.20798/) (2025)
- [SafeNeuron: Neuron-Level Safety Alignment for Large Language Models](https://mlsanity.com/paper/2602.12158/) (2026)
- [Safe Language Generation in the Limit](https://mlsanity.com/paper/2601.08648/) (2026)
- [Bridging AI and Software Security: A Comparative Vulnerability Assessment of LLM Agent Deployment Paradigms](https://mlsanity.com/paper/2507.06323/) (2025)
- [Large Language Model Sourcing: A Survey](https://mlsanity.com/paper/2510.10161/) (2025)
- [Eliciting and Analyzing Emergent Misalignment in State-of-the-Art Large Language Models](https://mlsanity.com/paper/2508.04196/) (2025)
- [Tone Matters: The Impact of Linguistic Tone on Hallucination in VLMs](https://mlsanity.com/paper/2601.06460/) (2026)
- [Adversarial Attacks on Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2603.27918/) (2026)
- [Trust Me, I Know This Function: Hijacking LLM Static Analysis using Bias](https://mlsanity.com/paper/2508.17361/) (2025)
- [LLM-as-Judge Framework for Evaluating Tone-Induced Hallucination in Vision-Language Models](https://mlsanity.com/paper/2604.18803/) (2026)
- [Countermind: A Multi-Layered Security Architecture for Large Language Models](https://mlsanity.com/paper/2510.11837/) (2025)
- [Zero-Shot Embedding Drift Detection: A Lightweight Defense Against Prompt Injections in LLMs](https://mlsanity.com/paper/2601.12359/) (2026)
- [The System Prompt Is the Attack Surface: How LLM Agent Configuration Shapes Security and Creates Exploitable Vulnerabilities](https://mlsanity.com/paper/2603.25056/) (2026)
- [Accelerating Scientific Research with Gemini in the Real-World](https://mlsanity.com/paper/2608.26701/) (2026)
- [Layered LLM Defenses as an Ensemble: Access Tiers, Inference Cost, and the Measured Failure Correlation Between Defense Layers](https://mlsanity.com/paper/2608.28327/) (2026)
- [Gecko: A Simulation Environment with Stateful Feedback for Refining Agent Tool Calls](https://mlsanity.com/paper/2602.19218/) (2026)
- [Stochasticity in Tokenisation Improves Robustness](https://mlsanity.com/paper/2604.16037/) (2026)
- [Stealthy Backdoor Attacks against LLMs Based on Natural Style Triggers](https://mlsanity.com/paper/2604.21700/) (2026)
- [The Curse of Helpfulness: Inverse Scaling Law in Robustness to Distractor Instructions via DistractionIF](https://mlsanity.com/paper/2605.29491/) (2026)
- [Small, Free, and Effective: Orchestrating Open-Weight Small Language Models to Outperform Single LLM for Malware Analysis](https://mlsanity.com/paper/2607.20216/) (2026)
- [Monotonicity as an Architectural Bias for Robust Language Models](https://mlsanity.com/paper/2602.02686/) (2026)
- [Integration of Object Detection and Small VLMs for Construction Safety Hazard Identification](https://mlsanity.com/paper/2604.05210/) (2026)
- [On the Privacy of LLMs: An Ablation Study](https://mlsanity.com/paper/2605.02255/) (2026)
- [APIOT: Autonomous Vulnerability Management Across Bare-Metal Industrial OT Networks](https://mlsanity.com/paper/2605.02346/) (2026)
- [On Understanding, Identifying, and Mitigating Vulnerabilities in Agentic Large Language Models](https://mlsanity.com/paper/2608.10530/) (2026)
- [MLCommons Jailbreak Benchmark v1.0](https://mlsanity.com/paper/2610.02827/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) such as ChatGPT and its competitors have caused a revolution in natural language processing, but their capabilities also introduce new security vulnerabilities. This survey provides a comprehensive overview of these emerging concerns, categorizing threats into several key areas: inference-time attacks via prompt manipulation; training-time attacks; misuse by malicious actors; and the inherent risks in autonomous LLM agents. Recently, a significant focus is increasingly being placed on the latter. We summarize recent academic and industrial studies from 2022 to 2025 that exemplify each threat, analyze existing defense mechanisms and their limitations, and identify open challenges in securing LLM-based applications. We conclude by emphasizing the importance of advancing robust, multi-layered security strategies to ensure LLMs are safe and beneficial.