mlsanityData to 7 Oct 2026

NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails

arXiv 2310.10501 ↗First posted 16 Oct 2023cs.CL cs.AIcited by 347

Not on the map. Why?

Cited by 140 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

NeMo Guardrails is an open-source toolkit for easily adding programmable guardrails to LLM-based conversational systems. Guardrails (or rails for short) are a specific way of controlling the output of an LLM, such as not talking about topics considered harmful, following a predefined dialogue path, using a particular language style, and more. There are several mechanisms that allow LLM providers and developers to add guardrails that are embedded into a specific model at training, e.g. using model alignment. Differently, using a runtime inspired from dialogue management, NeMo Guardrails allows developers to add programmable rails to LLM applications - these are user-defined, independent of the underlying LLM, and interpretable. Our initial results show that the proposed approach can be used with several LLM providers to develop controllable and safe LLM applications using programmable rails.

Key facts. Cited by 347 papers in the map (data through 7 Oct 2026). Builds on 28 of its 34 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

28 of 34 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  9. 2017Billion-scale similarity search with GPUscited by 2,477
  10. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  11. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  12. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
16 more
  1. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  2. 2022Red Teaming Language Models with Language Modelscited by 983
  3. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  4. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  5. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  6. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  7. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  8. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  9. 2022Prompting GPT-3 To Be Reliablecited by 258
  10. 2019Benchmarking Natural Language Understanding Services for building Conversational Agentscited by 166
  11. 2017Rasa: Open Source Language Understanding and Dialogue Managementcited by 67
  12. 2023Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Studycited by 63
  13. 2023SGP-TOD: Building Task Bots Effortlessly via Schema-Guided LLM Promptingcited by 39
  14. 2022Toxicity Detection with Generative Prompt-based Inferencecited by 35
  15. 2023Leveraging Implicit Feedback from Deployment Data in Dialoguecited by 14
  16. 2022Prompt Learning for Domain Adaptation in Task-Oriented Dialoguecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 78; 2025: 78; 2026: 187.

347 papers.

Sort:

By area:

  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  2. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 329
  3. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  4. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 150
  5. 2024Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferencescited by 142
  6. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 140
  7. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  8. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  9. 2025Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrailscited by 122
  10. 2025LlamaFirewall: An open source guardrail system for building secure AI agentscited by 114
  11. 2024GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoningcited by 113
  12. 2024Large Language Model Unlearning via Embedding-Corrupted Promptscited by 112
38 more
  1. 2023Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Surveycited by 92
  2. 2025GuardReasoner: Towards Reasoning-based LLM Safeguardscited by 89
  3. 2025AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detectioncited by 82
  4. 2024Safeguarding Large Language Models: A Surveycited by 73
  5. 2024RigorLLM: Resilient Guardrails for Large Language Models against Undesired Contentcited by 65
  6. 2024Building Guardrails for Large Language Modelscited by 61
  7. 2024Position: LLM Unlearning Benchmarks are Weak Measures of Progresscited by 58
  8. 2023Self-Guard: Empower the LLM to Safeguard Itselfcited by 52
  9. 2024garak: A Framework for Security Probing Large Language Modelscited by 48
  10. 2024Phantom: General Backdoor Attacks on Retrieval Augmented Language Generationcited by 46
  11. 2025VeriGuard: Enhancing LLM Agent Safety via Verified Code Generationcited by 46
  12. 2024Multicalibration for Confidence Scoring in LLMscited by 44
  13. 2024Current state of LLM Risks and AI Guardrailscited by 44
  14. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  15. 2024Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Modelscited by 42
  16. 2024R2-Guard: Robust Reasoning Enabled LLM Guardrail via Knowledge-Enhanced Logical Reasoningcited by 41
  17. 2024SafeWatch: An Efficient Safety-Policy Following Video Guardrail Model with Transparent Explanationscited by 37
  18. 2024MELTing point: Mobile Evaluation of Language Transformerscited by 35
  19. 2024Agent Design Pattern Catalogue: A Collection of Architectural Patterns for Foundation Model based Agentscited by 35
  20. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 31
  21. 2026Prompt Injection Attacks on Agentic Coding Assistants: A Systematic Analysis of Vulnerabilities in Skills, Tools, and Protocol Ecosystemscited by 30
  22. 2026From Agent Traces to Trust: A Survey of Evidence Tracing and Execution Provenance in LLM Agentscited by 29
  23. 2025Adversarial Reasoning at Jailbreaking Timecited by 27
  24. 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Surveycited by 26
  25. 2024Malla: Demystifying Real-world Large Language Model Integrated Malicious Servicescited by 24
  26. 2025Reliable Weak-to-Strong Monitoring of LLM Agentscited by 24
  27. 2024On Calibration of LLM-based Guard Models for Reliable Content Moderationcited by 21
  28. 2024Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey)cited by 20
  29. 2026AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systemscited by 20
  30. 2024SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelinescited by 19
  31. 2026Cordon: Semantic Transactions for Tool-Using LLM Agentscited by 19
  32. 2025Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasetscited by 18
  33. 2024Prompt Perturbation in Retrieval-Augmented Generation based Large Language Modelscited by 17
  34. 2024PROMPTFUZZ: Harnessing Fuzzing Techniques for Robust Testing of Prompt Injection in LLMscited by 17
  35. 2024SciSafeEval: A Comprehensive Benchmark for Safety Alignment of Large Language Models in Scientific Taskscited by 17
  36. 2025SoK: Evaluating Jailbreak Guardrails for Large Language Modelscited by 17
  37. 2026Intelligent AI Delegationcited by 16
  38. 2026A Framework for Formalizing LLM Agent Securitycited by 16

50 most cited of 347 All 347 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025No Free Lunch with Guardrails
  2. 2025Taxonomy-Adaptive Moderation Model with Robust Guardrails for Large Language Models
  3. 2025Safety Guardrails for LLM-Enabled Robots
  4. 2026PL-Guard: Probabilistic Logic Reasoning for LLM Guardrails
  5. 2025MrGuard: A Multilingual Reasoning Guardrail for Universal LLM Safety
  6. 2026Beyond Unsafe Detection: Counterfactually Anchored Evidence Attribution for Multi-Turn LLM Safety Failures
  7. 2025Safety Through Reasoning: An Empirical Study of Reasoning Guardrail Models
  8. 2026Do Safety Guardrails Need to Reason? LeanGuard: A Fast and Light Approach for Robust Moderation