# ControlNET: A Firewall for RAG-based LLM System

arXiv: https://arxiv.org/abs/2504.09593 · HTML: https://mlsanity.com/paper/2504.09593/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Privacy and security of RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 28 of its 51 references that are in the map.
- First posted on arXiv 13 Apr 2025.

## Placed in

- Retrieval-augmented generation / [Privacy and security of RAG](https://mlsanity.com/rag/privacy-and-security-of-rag/)
- LLM safety and security / [Membership inference attacks](https://mlsanity.com/safety/membership-inference-attacks/)

## Builds on (most cited first, 28 of 28)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022)
- [Extracting Training Data from Diffusion Models](https://mlsanity.com/paper/2301.13188/) (2023)
- [Prompt Injection attack against LLM-integrated Applications](https://mlsanity.com/paper/2306.05499/) (2023)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2405.06211/) (2024)
- [A Holistic Approach to Undesired Content Detection in the Real World](https://mlsanity.com/paper/2208.03274/) (2022)
- [Analyzing Leakage of Personally Identifiable Information in Language Models](https://mlsanity.com/paper/2302.00539/) (2023)
- [Optimization-based Prompt Injection Attack to LLM-as-a-Judge](https://mlsanity.com/paper/2403.17710/) (2024)
- [Attacks, Defenses and Evaluations for LLM Conversation Safety: A Survey](https://mlsanity.com/paper/2402.09283/) (2024)
- [PLeak: Prompt Leaking Attacks against Large Language Model Applications](https://mlsanity.com/paper/2405.06823/) (2024)
- [RigorLLM: Resilient Guardrails for Large Language Models against Undesired Content](https://mlsanity.com/paper/2403.13031/) (2024)
- [Assessing Prompt Injection Risks in 200+ Custom GPTs](https://mlsanity.com/paper/2311.11538/) (2023)
- [Prompt Stealing Attacks Against Text-to-Image Generation Models](https://mlsanity.com/paper/2302.09923/) (2023)
- [Prompt Stealing Attacks Against Large Language Models](https://mlsanity.com/paper/2402.12959/) (2024)
- [PromptCARE: Prompt Copyright Protection by Watermark Injection and Verification](https://mlsanity.com/paper/2308.02816/) (2023)
- [Sandwich attack: Multi-language Mixture Adaptive Attack on LLMs](https://mlsanity.com/paper/2404.07242/) (2024)
- [Raccoon: Prompt Extraction Benchmark of LLM-Integrated Applications](https://mlsanity.com/paper/2406.06737/) (2024)
- [Scaling Up LLM Reviews for Google Ads Content Moderation](https://mlsanity.com/paper/2402.14590/) (2024)
- [Multitask Mayhem: Unveiling and Mitigating Safety Gaps in LLMs Fine-tuning](https://mlsanity.com/paper/2409.15361/) (2024)
- [Eguard: Defending LLM Embeddings Against Inversion Attacks via Text Mutual Information Optimization](https://mlsanity.com/paper/2411.05034/) (2024)

## Built on by (most cited first, 12 of 12)

- [Silent Leaks: Implicit Knowledge Extraction Attack on RAG Systems through Benign Queries](https://mlsanity.com/paper/2505.15420/) (2025)
- [PromptSleuth: Detecting Prompt Injection via Semantic Intent Invariance](https://mlsanity.com/paper/2508.20890/) (2025)
- [Unveiling the Landscape of LLM Deployment in the Wild: An Empirical Study](https://mlsanity.com/paper/2505.02502/) (2025)
- [Quantifying Conversation Drift in MCP via Latent Polytope](https://mlsanity.com/paper/2508.06418/) (2025)
- [MemPot: Defending Against Memory Extraction Attack with Optimized Honeypots](https://mlsanity.com/paper/2602.07517/) (2026)
- [PRISM: Generation-Time Detection and Mitigation of Secret Leakage in Multi-Agent LLM Pipelines](https://mlsanity.com/paper/2605.10614/) (2026)
- [Red-Teaming Agent Execution Contexts: Open-World Security Evaluation on OpenClaw](https://mlsanity.com/paper/2605.11047/) (2026)
- [Safeguarding Multimodal Knowledge Copyright in the RAG-as-a-Service Environment](https://mlsanity.com/paper/2506.10030/) (2025)
- [Provably Secure Retrieval-Augmented Generation](https://mlsanity.com/paper/2508.01084/) (2025)
- [Black-Box Guardrail Reverse-engineering Attack](https://mlsanity.com/paper/2511.04215/) (2025)
- [Adaptive Defense Orchestration for RAG: A Sentinel-Strategist Architecture against Multi-Vector Attacks](https://mlsanity.com/paper/2604.20932/) (2026)
- [Adaptive Arena-based Contestable Argumentative Network-of-Experts for Open-Ended Care Plan Coordination](https://mlsanity.com/paper/2608.05391/) (2026)

## Abstract (arXiv)

Retrieval-Augmented Generation (RAG) has significantly enhanced the factual accuracy and domain adaptability of Large Language Models (LLMs). This advancement has enabled their widespread deployment across sensitive domains such as healthcare, finance, and enterprise applications. RAG mitigates hallucinations by integrating external knowledge, yet introduces privacy risk and security risk, notably data breaching risk and data poisoning risk. While recent studies have explored prompt injection and poisoning attacks, there remains a significant gap in comprehensive research on controlling inbound and outbound query flows to mitigate these threats. In this paper, we propose an AI firewall, ControlNET, designed to safeguard RAG-based LLM systems from these vulnerabilities. ControlNET controls query flows by leveraging activation shift phenomena to detect adversarial queries and mitigate their impact through semantic divergence. We conduct comprehensive experiments on four different benchmark datasets including Msmarco, HotpotQA, FinQA, and MedicalSys using state-of-the-art open source LLMs (Llama3, Vicuna, and Mistral). Our results demonstrate that ControlNET achieves over 0.909 AUROC in detecting and mitigating security threats while preserving system harmlessness. Overall, ControlNET offers an effective, robust, harmless defense mechanism, marking a significant advancement toward the secure deployment of RAG-based LLM systems.