mlsanityData to 7 Oct 2026

Safeguarding Large Language Models: A Survey

arXiv 2406.02622 ↗First posted 3 Jun 2024cs.CR cs.AIcited by 73

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

In the burgeoning field of Large Language Models (LLMs), developing a robust safety mechanism, colloquially known as "safeguards" or "guardrails", has become imperative to ensure the ethical use of LLMs within prescribed boundaries. This article provides a systematic literature review on the current status of this critical mechanism. It discusses its major challenges and how it can be enhanced into a comprehensive mechanism dealing with ethical issues in various contexts. First, the paper elucidates the current landscape of safeguarding mechanisms that major LLM service providers and the open-source community employ. This is followed by the techniques to evaluate, analyze, and enhance some (un)desirable properties that a guardrail might want to enforce, such as hallucinations, fairness, privacy, and so on. Based on them, we review techniques to circumvent these controls (i.e., attacks), to defend the attacks, and to reinforce the guardrails. While the techniques mentioned above represent the current status and the active research trends, we also discuss several challenges that cannot be easily dealt with by the methods and present our vision on how to implement a comprehensive guardrail through the full consideration of multi-disciplinary approach, neural-symbolic method, and systems development lifecycle.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 162 of its 215 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

162 of 215 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Building Guardrails for Large Language Models2 Feb
  2. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct

Other references · 160

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  10. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  11. 2016Deep Learning with Differential Privacycited by 2,682
  12. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
38 more
  1. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  2. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  3. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  4. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  5. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  6. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  7. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  8. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  9. 2023PaLM 2 Technical Reportcited by 1,154
  10. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  11. 2022Red Teaming Language Models with Language Modelscited by 983
  12. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  13. 2015Pointer Networkscited by 882
  14. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  15. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  16. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  17. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  18. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  19. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  20. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  21. 2023A Watermark for Large Language Modelscited by 667
  22. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  23. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  24. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  25. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  26. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
  27. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  28. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  29. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  30. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  31. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  32. 2023Detecting Language Model Attacks with Perplexitycited by 411
  33. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  34. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 401
  35. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  36. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  37. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  38. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 377

50 most cited of 160 All 160 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 35; 2026: 32.

73 papers.

Sort:

By area:

  1. 2025AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agentscited by 196
  2. 2024ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agentscited by 126
  3. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  4. 2025Building a Foundational Guardrail for General Agentic Systems via Synthetic Datacited by 26
  5. 2025Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directionscited by 25
  6. 2025Bypassing LLM Guardrails: An Empirical Analysis of Evasion Attacks against Prompt Injection and Jailbreak Detection Systemscited by 24
  7. 2025LLM-Powered AI Agent Systems and Their Applications in Industrycited by 22
  8. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 22
  9. 2024The Open Source Advantage in Large Language Models (LLMs)cited by 18
  10. 2026TrajAD: Trajectory Anomaly Detection for Trustworthy LLM Agentscited by 13
  11. 2025Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Modelscited by 11
  12. 2025POT: Inducing Overthinking in LLMs via Black-Box Iterative Optimizationcited by 11
38 more
  1. 2025Think Twice, Generate Once: Safeguarding by Progressive Self-Reflectioncited by 11
  2. 2025Measuring and Guiding Monosemanticitycited by 10
  3. 2025FairI Tales: Evaluation of Fairness in Indian Contexts with a Focus on Bias and Stereotypescited by 9
  4. 2025Enhancing Robustness of LLM-Driven Multi-Agent Systems through Randomized Smoothingcited by 9
  5. 2025Safety of Embodied Navigation: A Surveycited by 9
  6. 2025The TIP of the Iceberg: Revealing a Hidden Class of Task-in-Prompt Adversarial Attacks on LLMscited by 7
  7. 2025Shh, don't say that! Domain Certification in LLMscited by 7
  8. 2025Among Them: A game-based framework for assessing persuasion capabilities of LLMscited by 7
  9. 2025RedDebate: Safer Responses Through Multi-Agent Red Teaming Debatescited by 7
  10. 2025Stop Reducing Responsibility in LLM-Powered Multi-Agent Systems to Local Alignmentcited by 7
  11. 2026From Governance Norms to Enforceable Controls: A Layered Translation Method for Runtime Guardrails in Agentic AIcited by 7
  12. 2026Evolving Contextual Safety in Multi-Modal Large Language Models via Inference-Time Self-Reflective Memorycited by 5
  13. 2025ARACNE: An LLM-Based Autonomous Shell Pentesting Agentcited by 4
  14. 2026Beyond Simulations: What 20,000 Real Conversations Reveal About Mental Health AI Safetycited by 4
  15. 2024A Flexible Large Language Models Guardrail Development Methodology Applied to Off-Topic Prompt Detectioncited by 3
  16. 2025RAG Makes Guardrails Unsafe? Investigating Robustness of Guardrails under RAG-style Contextscited by 3
  17. 2026Multi-Modal Data-Enhanced Foundation Models for Prediction and Control in Wireless Networks: A Surveycited by 3
  18. 2026Responsible Agentic AI Requires Explicit Provenancecited by 3
  19. 2025Prompt Inject Detection with Generative Explanation as an Investigative Toolcited by 2
  20. 2025Bias Analysis and Mitigation through Protected Attribute Detection and Regard Classificationcited by 2
  21. 2025AdaptiveGuard: Towards Adaptive Runtime Safety for LLM-Powered Softwarecited by 2
  22. 2025Tailored untruths: How personalisation challenges LLM safeguardscited by 2
  23. 2025Generative AI in Sociological Research: State of the Disciplinecited by 2
  24. 2026Bielik Guard: Efficient Polish Language Safety Classifiers for LLM Content Moderationcited by 2
  25. 2026An Empirical Study of Multi-Generation Sampling for Jailbreak Detection in Large Language Modelscited by 2
  26. 2025FALCON: Fine-grained Activation Manipulation by Contrastive Orthogonal Unalignment for Large Language Modelcited by 1
  27. 2025The Tower of Babel Revisited: Multilingual Jailbreak Prompts on Closed-Source Large Language Modelscited by 1
  28. 2025Customize Multi-modal RAI Guardrails with Precedent-based predictionscited by 1
  29. 2025Echoes of Human Malice in Agents: Benchmarking LLMs for Multi-Turn Online Harassment Attackscited by 1
  30. 2026ElecTwit: A Framework for Studying Persuasion in Multi-Agent Social Systemscited by 1
  31. 2026Step-Wise Refusal Dynamics in Autoregressive and Diffusion Language Modelscited by 1
  32. 2026A Content-Based Framework for Cybersecurity Refusal Decisions in Large Language Modelscited by 1
  33. 2026Guardian-as-an-Advisor: Advancing Next-Generation Guardian Models for Trustworthy LLMscited by 1
  34. 2026Don't Make Models Guess Security and Safety: Symbolic Guardrails for Domain-Specific AI Agentscited by 1
  35. 2026Do LLMs have core beliefs?cited by 1
  36. 2026SafePyramid: A Hierarchical Benchmark for In-context Policy Guardrailingcited by 1
  37. 2024Knowledge Return Oriented Prompting (KROP)cited by 0
  38. 2024Standardization Trends on Safety and Trustworthiness Technology for Advanced AIcited by 0

50 most cited of 73 All 73 (JSON)