mlsanityData to 7 Oct 2026

RAG Makes Guardrails Unsafe? Investigating Robustness of Guardrails under RAG-style Contexts

arXiv 2510.05310 ↗First posted 6 Oct 2025cs.CL cs.AIcited by 3

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

With the increasing adoption of large language models (LLMs), ensuring the safety of LLM systems has become a pressing concern. External LLM-based guardrail models have emerged as a popular solution to screen unsafe inputs and outputs, but they are themselves fine-tuned or prompt-engineered LLMs that are vulnerable to data distribution shifts. In this paper, taking Retrieval Augmentation Generation (RAG) as a case study, we investigated how robust LLM-based guardrails are against additional information embedded in the context. Through a systematic evaluation of 3 Llama Guards and 2 GPT-oss models, we confirmed that inserting benign documents into the guardrail context alters the judgments of input and output guardrails in around 11% and 8% of cases, making them unreliable. We separately analyzed the effect of each component in the augmented context: retrieved documents, user query, and LLM-generated response. The two mitigation methods we tested only bring minor improvements. These results expose a context-robustness gap in current guardrails and motivate training and evaluation protocols that are robust to retrieval and query composition.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 38 of its 43 references that are in the map. First posted on arXiv 6 Oct 2025.

Builds on

38 of 43 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Models25 Apr
  2. 2024R2-Guard: Robust Reasoning Enabled LLM Guardrail via Knowledge-Enhanced Logical Reasoning8 Jul
  3. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs26 Jun
  4. 2024Safeguarding Large Language Models: A Survey3 Jun
  5. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Models31 May

Other references · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024GPT-4o System Cardcited by 4,467
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  7. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  8. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  9. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  10. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  11. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  12. 2024A Survey on Large Language Models for Code Generationcited by 646
21 more
  1. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  2. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  3. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
  4. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  5. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 251
  6. 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 224
  7. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 213
  8. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 191
  9. 2024The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)cited by 159
  10. 2024AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Expertscited by 126
  11. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 105
  12. 2024BM25S: Orders of magnitude faster lexical search via eager sparse scoringcited by 87
  13. 2024BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Modelscited by 84
  14. 2023Attack Prompt Generation for Red Teaming and Defending Large Language Modelscited by 73
  15. 2024TrojanRAG: Retrieval-Augmented Generation Can Be Backdoor Driver in Large Language Modelscited by 63
  16. 2023AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLM-powered Applicationscited by 53
  17. 2025SafeRAG: Benchmarking Security in Retrieval-Augmented Generation of Large Language Modelcited by 31
  18. 2025Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Surveycited by 29
  19. 2024Does RAG Introduce Unfairness in LLMs? Evaluating Fairness in Retrieval-Augmented Generation Systemscited by 26
  20. 2025How to Protect Yourself from 5G Radiation? Investigating LLM Responses to Implicit Misinformationcited by 17
  21. 2025FairSense: Long-Term Fairness Analysis of ML-Enabled Systemscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Towards Secure Retrieval-Augmented Generation: A Comprehensive Review of Threats, Defenses and Benchmarkscited by 3
  2. 2026Compliance-Scored Best-of-N Guardrail Orchestration for Multimodal Document Generation in Payments Dispute Defensecited by 1
  3. 2026Relevance as a Vulnerability: How Web Retrieval Degrades Safety Alignment in LLM Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026RAG-Safety-Bench: Reliable Evaluation of Retrieval-Augmented LLM Safety
  2. 2025Evaluating the Retrieval Robustness of Large Language Models
  3. 2026Retrieved But Not Reliable: A Survey on Attacks, and Defenses in Retrieval-Augmented Generation
  4. 2024Typos that Broke the RAG's Back: Genetic Attack on RAG Pipeline by Simulating Documents in the Wild via Low-level Perturbations
  5. 2024Current state of LLM Risks and AI Guardrails
  6. 2025RAG Security and Privacy: Formalizing the Threat Model and Attack Surface
  7. 2026When Safety Becomes a Vulnerability: Exploiting LLM Alignment Homogeneity for Transferable Blocking in RAG
  8. 2024Long Context RAG Performance of Large Language Models