mlsanityData to 14 Oct 2026

LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systems

arXiv 2509.10682 ↗First posted 12 Sep 2025cs.CR cs.AI cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

The success and wide adoption of generative AI (GenAI), particularly large language models (LLMs), has attracted the attention of cybercriminals seeking to abuse models, steal sensitive data, or disrupt services. Moreover, providing security to LLM-based systems is a great challenge, as both traditional threats to software applications and threats targeting LLMs and their integration must be mitigated. In this survey, we shed light on security and privacy concerns of such LLM-based systems by performing a systematic review and comprehensive categorization of threats and defensive strategies considering the entire software and LLM life cycles. We analyze real-world scenarios with distinct characteristics of LLM usage, spanning from development to operation. In addition, threats are classified according to their severity level and to which scenarios they pertain, facilitating the identification of the most relevant threats. Recommended defense strategies are systematically categorized and mapped to the corresponding life cycle phase and possible attack strategies they attenuate. This work paves the way for consumers and vendors to understand and efficiently mitigate risks during integration of LLMs in their respective solutions or organizations. It also enables the research community to benefit from the discussion of open challenges and edge cases that may hinder the secure and privacy-preserving adoption of LLM-based systems.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 239 of its 346 references that are in the map. First posted on arXiv 12 Sep 2025.

Builds on

239 of 346 references.

References · 239

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  9. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  10. 2020Extracting Training Data from Large Language Modelscited by 2,189
  11. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  12. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
38 more
  1. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  2. 2022Red Teaming Language Models with Language Modelscited by 1,026
  3. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  4. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  5. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 678
  6. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  7. 2023Prompt Injection attack against LLM-integrated Applicationscited by 605
  8. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  9. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  10. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 498
  11. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  12. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  13. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  14. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 437
  15. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 429
  16. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 390
  17. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  18. 2023Who's Harry Potter? Approximate Unlearning in LLMscited by 381
  19. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 364
  20. 2017Deep Models Under the GAN: Information Leakage from Collaborative Deep Learningcited by 334
  21. 2023Low-Resource Languages Jailbreak GPT-4cited by 309
  22. 2023Role-Play with Large Language Modelscited by 294
  23. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
  24. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 264
  25. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 263
  26. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  27. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 259
  28. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 258
  29. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 230
  30. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Modelscited by 223
  31. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Triggercited by 222
  32. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMscited by 215
  33. 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMscited by 199
  34. 2021Evaluating Gradient Inversion Attacks and Defenses in Federated Learningcited by 196
  35. 2023Text Embeddings Reveal (Almost) As Much As Textcited by 192
  36. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 186
  37. 2023AutoDroid: LLM-powered Task Automation in Androidcited by 184
  38. 2023MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbotscited by 183

50 most cited of 239 All 239 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Do Multimodal RAG Systems Leak Data? A Comprehensive Evaluation of Membership Inference and Image Caption Retrieval Attackscited by 3