mlsanityData to 7 Oct 2026

Building Guardrails for Large Language Models

arXiv 2402.01822 ↗First posted 2 Feb 2024cs.CL cs.AIcited by 61

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

As Large Language Models (LLMs) become more integrated into our daily lives, it is crucial to identify and mitigate their risks, especially when the risks can have profound impacts on human users and societies. Guardrails, which filter the inputs or outputs of LLMs, have emerged as a core safeguarding technology. This position paper takes a deep look at current open-source solutions (Llama Guard, Nvidia NeMo, Guardrails AI), and discusses the challenges and the road towards building more complete solutions. Drawing on robust evidence from previous research, we advocate for a systematic approach to construct guardrails for LLMs, based on comprehensive consideration of diverse contexts across various LLMs applications. We propose employing socio-technical methods through collaboration with a multi-disciplinary team to pinpoint precise technical requirements, exploring advanced neural-symbolic implementations to embrace the complexity of the requirements, and developing verification and testing to ensure the utmost quality of the final product.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 113 of its 148 references that are in the map. First posted on arXiv 2 Feb 2024.

Builds on

113 of 148 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Robust Safety Classifier for Large Language Models: Adversarial Prompt Shield31 Oct

Other references · 112

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  4. 2016Deep Learning with Differential Privacycited by 2,682
  5. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  6. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  7. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  8. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  9. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  10. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  11. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  12. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
38 more
  1. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  2. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  3. 2022Red Teaming Language Models with Language Modelscited by 983
  4. 2022Mass-Editing Memory in a Transformercited by 976
  5. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  6. 2015Pointer Networkscited by 882
  7. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  8. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  9. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  10. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  11. 2023A Watermark for Large Language Modelscited by 667
  12. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  13. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  14. 2020On Adaptive Attacks to Adversarial Example Defensescited by 547
  15. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  16. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  17. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  18. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  19. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  20. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 376
  21. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
  22. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 329
  23. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  24. 2023Chain-of-Verification Reduces Hallucination in Large Language Modelscited by 317
  25. 2023Low-Resource Languages Jailbreak GPT-4cited by 290
  26. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 286
  27. 2016Adversarial Training Methods for Semi-Supervised Text Classificationcited by 280
  28. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  29. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  30. 2023How is ChatGPT's behavior changing over time?cited by 265
  31. 2016Safety Verification of Deep Neural Networkscited by 262
  32. 2023Certifying LLM Safety against Adversarial Promptingcited by 248
  33. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 247
  34. 2019The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networkscited by 241
  35. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 218
  36. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 195
  37. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 191
  38. 2023Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMscited by 191

50 most cited of 112 All 112 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 25; 2026: 20.

61 papers.

Sort:

By area:

  1. 2024Large Language Model Unlearning via Embedding-Corrupted Promptscited by 112
  2. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  3. 2025LLMs Encode Harmfulness and Refusal Separatelycited by 75
  4. 2024Safeguarding Large Language Models: A Surveycited by 73
  5. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 31
  6. 2025LLM-Powered AI Agent Systems and Their Applications in Industrycited by 22
  7. 2025The Sum Leaks More Than Its Parts: Compositional Privacy Risks and Mitigations in Multi-Agent Collaborationcited by 19
  8. 2025Enhancing LLM Safety Through a Theoretical Minimax Game Lenscited by 18
  9. 2025DynaGuard: A Dynamic Guardian Model With User-Defined Policiescited by 16
  10. 2026Intelligent AI Delegationcited by 16
  11. 2025Refining Input Guardrails: Enhancing LLM-as-a-Judge Efficiency Through Chain-of-Thought Fine-Tuning and Alignmentcited by 14
  12. 2024Swiss Cheese Model for AI Safety: A Taxonomy and Reference Architecture for Multi-Layered Guardrails of Foundation Model Based Agentscited by 13
38 more
  1. 2026Agent-Sentry: Bounding LLM Agents via Execution Provenancecited by 13
  2. 2025Policy-as-Prompt: Turning AI Governance Rules into Guardrails for AI Agentscited by 12
  3. 2024The Need for Guardrails with Large Language Models in Medical Safety-Critical Settings: An Artificial Intelligence Application in the Pharmacovigilance Ecosystemcited by 11
  4. 2024Trust-Oriented Adaptive Guardrails for Large Language Modelscited by 8
  5. 2024Tiny Refinements Elicit Resilience: Toward Efficient Prefix-Model Against LLM Red-Teamingcited by 6
  6. 2024Building a Domain-specific Guardrail Model in Productioncited by 6
  7. 2025LLM Content Moderation and User Satisfaction: Evidence from Response Refusals in Chatbot Arenacited by 6
  8. 2026When Bots Take the Bait: Exposing and Mitigating the Emerging Social Engineering Attack in Web Automation Agentcited by 6
  9. 2026Sparse Models, Sparse Safety: Unsafe Routes in Mixture-of-Experts LLMscited by 6
  10. 2024LLM-Agent-UMF: LLM-based Agent Unified Modeling Framework for Seamless Design of Multi Active/Passive Core-Agent Architecturescited by 5
  11. 2025No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanismscited by 5
  12. 2025Efficient LLM Moderation with Multi-Layer Latent Prototypescited by 4
  13. 2025LM Agents May Fail to Act on Their Own Risk Knowledgecited by 4
  14. 2025Co-Investigator AI: The Rise of Agentic AI for Smarter, Trustworthy AML Compliance Narrativescited by 4
  15. 2025Domain-Specific Constitutional AI: Enhancing Safety in LLM-Powered Mental Health Chatbotscited by 4
  16. 2024Layer-wise Alignment: Examining Safety Alignment Across Image Encoder Layers in Vision Language Modelscited by 3
  17. 2025Engineering LLM Powered Multi-agent Framework for Autonomous CloudOpscited by 3
  18. 2025Transduction is All You Need for Structured Data Workflowscited by 3
  19. 2025Lie to Me: Knowledge Graphs for Robust Hallucination Self-Detection in LLMscited by 3
  20. 2026Skill Drift Is Contract Violation: Proactive Maintenance for LLM Agent Skill Librariescited by 3
  21. 2025Prompt Inject Detection with Generative Explanation as an Investigative Toolcited by 2
  22. 2025ConceptGuard: Neuro-Symbolic Safety Guardrails via Sparse Interpretable Jailbreak Conceptscited by 2
  23. 2025AdvJudge-Zero: Binary Decision Flips in LLM-as-a-Judge via Adversarial Control Tokenscited by 2
  24. 2024Guardrails for avoiding harmful medical product recommendations and off-label promotion in generative AI modelscited by 1
  25. 2024AI Ethics by Design: Implementing Customizable Guardrails for Responsible AI Developmentcited by 1
  26. 2025Is It Time To Treat Prompts As Code? A Multi-Use Case Study For Prompt Optimization Using DSPycited by 1
  27. 2025Customize Multi-modal RAI Guardrails with Precedent-based predictionscited by 1
  28. 2025From Refusal to Recovery: A Control-Theoretic Approach to Generative AI Guardrailscited by 1
  29. 2026TRYLOCK: Defense-in-Depth Against LLM Jailbreaks via Layered Preference and Representation Engineeringcited by 1
  30. 2026Greater accessibility can amplify discrimination in generative AIcited by 1
  31. 2026Trustworthy Agent Network: Trust in Agent Networks Must Be Baked In, Not Bolted Oncited by 1
  32. 2026Safety Beyond the Interface: Detecting Harm via Latent States in Large Language Modelscited by 1
  33. 2024SUKHSANDESH: An Avatar Therapeutic Question Answering Platform for Sexual Education in Rural Indiacited by 0
  34. 2024Building Understandable Messaging for Policy and Evidence Review (BUMPER) with AIcited by 0
  35. 2024"Moralized" Multi-Step Jailbreak Prompts: Black-Box Testing of Guardrails in Large Language Models for Verbal Attackscited by 0
  36. 2024Considerations Influencing Offense-Defense Dynamics From Artificial Intelligencecited by 0
  37. 2025Policy alone is probably not the solution: A large-scale experiment on how developers struggle to design meaningful end-user explanationscited by 0
  38. 2025Conversational No-code, Multi-agentic Disease Module Identification and Drug Repurposing Prediction with ChatDRexcited by 0

50 most cited of 61 All 61 (JSON)