mlsanityData to 14 Oct 2026

A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment

arXiv 2504.15585 ↗First posted 22 Apr 2025cs.CR cs.AI cs.CLcited by 116

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 2 more
Abstract (arXiv)

The remarkable success of Large Language Models (LLMs) has illuminated a promising pathway toward achieving Artificial General Intelligence for both academic and industrial communities, owing to their unprecedented performance across various applications. As LLMs continue to gain prominence in both research and commercial domains, their security and safety implications have become a growing concern, not only for researchers and corporations but also for every nation. Currently, existing surveys on LLM safety primarily focus on specific stages of the LLM lifecycle, e.g., deployment phase or fine-tuning phase, lacking a comprehensive understanding of the entire "lifechain" of LLMs. To address this gap, this paper introduces, for the first time, the concept of "full-stack" safety to systematically consider safety issues throughout the entire process of LLM training, deployment, and eventual commercialization. Compared to the off-the-shelf LLM safety surveys, our work demonstrates several distinctive advantages: (I) Comprehensive Perspective. We define the complete LLM lifecycle as encompassing data preparation, pre-training, post-training, deployment and final commercialization. To our knowledge, this represents the first safety survey to encompass the entire lifecycle of LLMs. (II) Extensive Literature Support. Our research is grounded in an exhaustive review of over 800+ papers, ensuring comprehensive coverage and systematic organization of security issues within a more holistic understanding. (III) Unique Insights. Through systematic literature analysis, we have developed reliable roadmaps and perspectives for each chapter. Our work identifies promising research directions, including safety in data generation, alignment techniques, model editing, and LLM-based agent systems. These insights provide valuable guidance for researchers pursuing future work in this field.

Key facts. Cited by 116 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 555 of its 968 references that are in the map. First posted on arXiv 22 Apr 2025.

Builds on

555 of 968 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems5 Mar
  2. 2025Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyond26 Feb
  3. 2025SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities17 Feb
  4. 2025Nuclear Deployed: Analyzing Catastrophic Risks in Decision-making of Autonomous LLM Agents17 Feb
  5. 2025GuardReasoner: Towards Reasoning-based LLM Safeguards30 Jan
  6. 2024OpenAI o1 System Card21 Dec
  7. 2024Deliberative Alignment: Reasoning Enables Safer Language Models20 Dec
  8. 2024Unified Parameter-Efficient Unlearning for LLMs30 Nov
  9. 2024AFlow: Automating Agentic Workflow Generation14 Oct
  10. 2024Sibyl: Simple yet Effective Agent Framework for Complex Real-world Reasoning15 Jul
  11. 2024GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning13 Jun
  12. 2024Learning to Poison Large Language Models for Downstream Manipulation21 Feb
2 more
  1. 2023Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinking16 Nov
  2. 2023Reasoning with Language Model is Planning with World Model24 May

Other references · 541

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2024GPT-4o System Cardcited by 4,588
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  10. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  11. 2023Qwen Technical Reportcited by 3,344
  12. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
38 more
  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  4. 2023A Survey of Large Language Modelscited by 2,576
  5. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  8. 2016Concrete Problems in AI Safetycited by 1,642
  9. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  10. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  11. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  12. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  13. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  14. 2023PaLM 2 Technical Reportcited by 1,162
  15. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  16. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  17. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  18. 2022Editing Models with Task Arithmeticcited by 1,120
  19. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,063
  20. 2022Red Teaming Language Models with Language Modelscited by 1,026
  21. 2022Mass-Editing Memory in a Transformercited by 1,001
  22. 2023ChatDev: Communicative Agents for Software Developmentcited by 919
  23. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  24. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  25. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 890
  26. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
  27. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  28. 2024DeepSeek-V3 Technical Reportcited by 811
  29. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
  30. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  31. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  32. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 769
  33. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  34. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  35. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  36. 2024Yi: Open Foundation Models by 01.AIcited by 719
  37. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  38. 2023Baichuan 2: Open Large-scale Language Modelscited by 670

50 most cited of 541 All 541 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 62; 2026: 54.

116 papers.

Sort:

By area:

  1. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
  2. 2025From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discoverycited by 77
  3. 2025TRiSM for Agentic AI: A Review of Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systemscited by 59
  4. 2025From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflowscited by 58
  5. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 57
  6. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  7. 2025AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraintcited by 49
  8. 2025Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AIcited by 42
  9. 2025Advancing Expert Specialization for Better MoEcited by 39
  10. 2025AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructionscited by 38
  11. 2025Look Back to Reason Forward: Revisitable Memory for Long-Context LLM Agentscited by 38
  12. 2025Building a Foundational Guardrail for General Agentic Systems via Synthetic Datacited by 29
38 more
  1. 2025We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systemscited by 27
  2. 2025Jailbreaking Large Language Diffusion Models: Revealing Hidden Safety Flaws in Diffusion-Based Text Generationcited by 26
  3. 2025LLM-based Agentic Reasoning Frameworks: A Survey from Methods to Scenarioscited by 26
  4. 2025FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought Reasoningcited by 26
  5. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 25
  6. 2025Get Experience from Practice: LLM Agents with Record & Replaycited by 23
  7. 2025FreezeVLA: Action-Freezing Attacks against Vision-Language-Action Modelscited by 21
  8. 2025Beyond the Protocol: Unveiling Attack Vectors in the Model Context Protocol (MCP) Ecosystemcited by 19
  9. 2025Oyster-I: Beyond Refusal -- Constructive Safety Alignment for Responsible Language Modelscited by 19
  10. 2025False Sense of Security: Why Probing-based Malicious Input Detection Fails to Generalizecited by 19
  11. 2026Temporary Authority, Permanent Effects: Commit-Time Authorization for LLM Agentscited by 18
  12. 2025Security of Internet of Agents: Attacks and Countermeasurescited by 17
  13. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  14. 2025MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Modelscited by 17
  15. 2025SafeMobile: Chain-level Jailbreak Detection and Automated Evaluation for Multimodal Mobile Agentscited by 15
  16. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 15
  17. 2025Graph-Augmented Large Language Model Agents: Current Progress and Future Prospectscited by 14
  18. 2025A Vision for Access Control in LLM-based Agent Systemscited by 14
  19. 2025An Empirical Study of Agent Developer Practices in AI Agent Frameworkscited by 14
  20. 2026When Lower Privileges Suffice: Investigating Over-Privileged Tool Selection in LLM Agentscited by 13
  21. 2025From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystemcited by 12
  22. 2025SafeSieve: From Heuristics to Experience in Progressive Pruning for LLM-based Multi-Agent Communicationcited by 12
  23. 2025Check Yourself Before You Wreck Yourself: Selectively Quitting Improves LLM Agent Safetycited by 12
  24. 2025Self-Jailbreaking: Language Models Can Reason Themselves Out of Safety Alignment After Benign Reasoning Trainingcited by 12
  25. 2025Backdoor Attribution: Elucidating and Controlling Backdoor in Language Modelscited by 10
  26. 2026Train at Moving Edge: Online-Verified Prompt Selection for Efficient RL Training of Large Reasoning Modelcited by 10
  27. 2025The Eye of Sherlock Holmes: Uncovering User Private Attribute Profiling via Vision-Language Model Agentic Frameworkcited by 9
  28. 2025Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Modelscited by 9
  29. 2026Drift-Bench: Diagnosing Cooperative Breakdowns in LLM Agents under Input Faults via Multi-Turn Interactioncited by 9
  30. 2026ProbeLLM: Automating Principled Diagnosis of LLM Failurescited by 9
  31. 2025Three Minds, One Legend: Jailbreak Large Reasoning Model with Adaptive Stacked Cipherscited by 8
  32. 2025Goal-Aware Identification and Rectification of Misinformation in Multi-Agent Systemscited by 8
  33. 2025Hidden in the Noise: Unveiling Backdoors in Audio LLMs Alignment through Latent Acoustic Pattern Triggerscited by 8
  34. 2025Evolve the Method, Not the Prompts: Evolutionary Synthesis of Jailbreak Attacks on LLMscited by 8
  35. 2026Risky-Bench: Probing Agentic Safety Risks under Real-World Deploymentcited by 8
  36. 2026AgentDoG 1.5: A Lightweight and Scalable Alignment Framework for AI Agent Safety and Securitycited by 8
  37. 2025Practical Reasoning Interruption Attacks on Reasoning Large Language Modelscited by 7
  38. 2025CCHall: A Novel Benchmark for Joint Cross-Lingual and Cross-Modal Hallucinations Detection in Large Language Modelscited by 7

50 most cited of 116 All 116 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025OffTopicEval: When Large Language Models Enter the Wrong Chat, Almost Always!
  2. 2025SafetyFlow: An Agent-Flow System for Automated LLM Safety Benchmarking
  3. 2024Current state of LLM Risks and AI Guardrails
  4. 2024SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types
  5. 2024AI Safety in Generative AI Large Language Models: A Survey
  6. 2025Measuring What Matters: A Framework for Evaluating Safety Risks in Real-World LLM Applications
  7. 2025The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating It
  8. 2025Is Safety Standard Same for Everyone? User-Specific Safety Evaluation of Large Language Models