mlsanityData to 14 Oct 2026

Data Security in Large Language Models: Risks, Defense, and Directions

arXiv 2508.02312 ↗First posted 4 Aug 2025cs.CR cs.AIcited by 0

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Large Language Models (LLMs), now a foundation in advancing natural language processing, power applications such as text generation, machine translation, and conversational systems. Despite their transformative potential, these models inherently rely on massive amounts of training data, often collected from diverse and uncurated sources, which exposes them to serious data security risks. Harmful or malicious data can compromise model behavior, leading to toxic outputs or hallucinations, while also creating vulnerabilities to data-driven attacks such as prompt injection and data poisoning. As LLMs continue to be integrated into critical real-world systems, understanding and addressing these data-centric security risks is imperative to safeguard user trust and system reliability. This survey offers a comprehensive overview of the main data security risks facing LLMs and reviews current defense strategies, including adversarial training, data cleaning, output guardrails, Reinforcement Learning from Human Feedback (RLHF), data augmentation, and Retrieval-Augmented Generation (RAG)/agent defenses. Additionally, we categorize and analyze relevant datasets used for assessing robustness and security across different domains, providing guidance for future research. Finally, we highlight key research directions that focus on data provenance and traceability, verifiable machine forgetting, secure model updates, standardized evaluation framework, explainability-driven security analysis, and effective governance frameworks, aiming to promote the safe and responsible development of LLM technology. This work seeks to inform researchers, practitioners, and policymakers, driving progress toward data security in LLMs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 70 of its 121 references that are in the map. First posted on arXiv 4 Aug 2025.

Builds on

70 of 121 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases17 Jul
  2. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judge26 Mar
  3. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents5 Mar
  4. 2024Security and Privacy Challenges of Large Language Models: A Survey30 Jan
  5. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb

Other references · 65

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  7. 2016Learning without Forgettingcited by 2,496
  8. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  9. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  10. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  11. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  12. 2023Extracting Training Data from Diffusion Modelscited by 732
38 more
  1. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  2. 2023Bias and Fairness in Large Language Models: A Surveycited by 639
  3. 2020On Adaptive Attacks to Adversarial Example Defensescited by 555
  4. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  5. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  6. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  7. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 452
  8. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  9. 2020Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditingcited by 384
  10. 2020Weight Poisoning Attacks on Pre-trained Modelscited by 341
  11. 2017Certified Defenses for Data Poisoning Attackscited by 303
  12. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 290
  13. 2023Poisoning Language Models During Instruction Tuningcited by 269
  14. 2018Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learningcited by 225
  15. 2021Trustworthy AI: From Principles to Practicescited by 207
  16. 2023Inverse Scaling: When Bigger Isn't Bettercited by 207
  17. 2021Societal Biases in Language Generation: Progress and Challengescited by 202
  18. 2023Beyond Memorization: Violating Privacy Via Inference with Large Language Modelscited by 175
  19. 2024The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)cited by 159
  20. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injectioncited by 157
  21. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attackscited by 145
  22. 2025Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrailscited by 132
  23. 2023On the Risk of Misinformation Pollution with Large Language Modelscited by 128
  24. 2024PLeak: Prompt Leaking Attacks against Large Language Model Applicationscited by 127
  25. 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Modelscited by 106
  26. 2019It's All in the Name: Mitigating Gender Bias with Name-Based Counterfactual Data Substitutioncited by 103
  27. 2024On Protecting the Data Privacy of Large Language Models (LLMs): A Surveycited by 96
  28. 2024BadEdit: Backdooring large language models by model editingcited by 96
  29. 2023Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Modelscited by 83
  30. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Surveycited by 83
  31. 2024Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systemscited by 76
  32. 2024SafeBench: A Safety Evaluation Framework for Multimodal Large Language Modelscited by 69
  33. 2024BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Modelscited by 50
  34. 2018Detection of Adversarial Training Examples in Poisoning Attacks through Anomaly Detectioncited by 46
  35. 2024On Large Language Models' Hallucination with Regard to Known Factscited by 43
  36. 2024A Survey of Recent Backdoor Attacks and Defenses in Large Language Modelscited by 36
  37. 2025Red Teaming the Mind of the Machine: A Systematic Evaluation of Prompt Injection and Jailbreak Vulnerabilities in LLMscited by 34
  38. 2024PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoningcited by 33

50 most cited of 65 All 65 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Models
  2. 2025Attack and defense techniques in large language models: A survey and new perspectives
  3. 2025A Survey of Attacks on Large Language Models
  4. 2024Recent Advances in Attack and Defense Approaches of Large Language Models
  5. 2024Exploring Vulnerabilities and Protections in Large Language Models: A Survey
  6. 2024Unique Security and Privacy Threats of Large Language Models: A Comprehensive Survey
  7. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks
  8. 2025Security Concerns for Large Language Models: A Survey