ControlNET: A Firewall for RAG-based LLM System
On the map: Retrieval-augmented generation → Privacy and security of RAG
also in 1 more
Abstract (arXiv)
Retrieval-Augmented Generation (RAG) has significantly enhanced the factual accuracy and domain adaptability of Large Language Models (LLMs). This advancement has enabled their widespread deployment across sensitive domains such as healthcare, finance, and enterprise applications. RAG mitigates hallucinations by integrating external knowledge, yet introduces privacy risk and security risk, notably data breaching risk and data poisoning risk. While recent studies have explored prompt injection and poisoning attacks, there remains a significant gap in comprehensive research on controlling inbound and outbound query flows to mitigate these threats. In this paper, we propose an AI firewall, ControlNET, designed to safeguard RAG-based LLM systems from these vulnerabilities. ControlNET controls query flows by leveraging activation shift phenomena to detect adversarial queries and mitigate their impact through semantic divergence. We conduct comprehensive experiments on four different benchmark datasets including Msmarco, HotpotQA, FinQA, and MedicalSys using state-of-the-art open source LLMs (Llama3, Vicuna, and Mistral). Our results demonstrate that ControlNET achieves over 0.909 AUROC in detecting and mitigating security threats while preserving system harmlessness. Overall, ControlNET offers an effective, robust, harmless defense mechanism, marking a significant advancement toward the secure deployment of RAG-based LLM systems.
Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Privacy and security of RAG, one of 15 directions in Retrieval-augmented generation. Builds on 28 of its 51 references that are in the map. First posted on arXiv 13 Apr 2025.
Builds on
28 of 51 references.
References · 28
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
- 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
- 2020Extracting Training Data from Large Language Modelscited by 2,144
- 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
- 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
- 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
- 2023Extracting Training Data from Diffusion Modelscited by 723
- 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
- 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
16 more
- 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
- 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
- 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
- 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
- 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 123
- 2024PLeak: Prompt Leaking Attacks against Large Language Model Applicationscited by 113
- 2024RigorLLM: Resilient Guardrails for Large Language Models against Undesired Contentcited by 65
- 2023Assessing Prompt Injection Risks in 200+ Custom GPTscited by 49
- 2023Prompt Stealing Attacks Against Text-to-Image Generation Modelscited by 48
- 2024Prompt Stealing Attacks Against Large Language Modelscited by 39
- 2023PromptCARE: Prompt Copyright Protection by Watermark Injection and Verificationcited by 30
- 2024Sandwich attack: Multi-language Mixture Adaptive Attack on LLMscited by 20
- 2024Raccoon: Prompt Extraction Benchmark of LLM-Integrated Applicationscited by 16
- 2024Scaling Up LLM Reviews for Google Ads Content Moderationcited by 12
- 2024Multitask Mayhem: Unveiling and Mitigating Safety Gaps in LLMs Fine-tuningcited by 8
- 2024Eguard: Defending LLM Embeddings Against Inversion Attacks via Text Mutual Information Optimizationcited by 8
Built on by
Citing papers per year: 2025: 6; 2026: 5.
11 papers.
- 2025Silent Leaks: Implicit Knowledge Extraction Attack on RAG Systems through Benign Queriescited by 19
- 2025Unveiling the Landscape of LLM Deployment in the Wild: An Empirical Studycited by 4
- 2025Quantifying Conversation Drift in MCP via Latent Polytopecited by 2
- 2026MemPot: Defending Against Memory Extraction Attack with Optimized Honeypotscited by 2
- 2026PRISM: Generation-Time Detection and Mitigation of Secret Leakage in Multi-Agent LLM Pipelinescited by 2
- 2026Red-Teaming Agent Execution Contexts: Open-World Security Evaluation on OpenClawcited by 2
- 2025Safeguarding Multimodal Knowledge Copyright in the RAG-as-a-Service Environmentcited by 1
- 2025Provably Secure Retrieval-Augmented Generationcited by 0
- 2025Black-Box Guardrail Reverse-engineering Attackcited by 0
- 2026Adaptive Defense Orchestration for RAG: A Sentinel-Strategist Architecture against Multi-Vector Attackscited by 0
- 2026Adaptive Arena-based Contestable Argumentative Network-of-Experts for Open-Ended Care Plan Coordinationcited by 0
- 2026Adaptive Arena-based Contestable Argumentative Network-of-Experts for Open-Ended Care Plan Coordination5 Aug 2026
- 2026Red-Teaming Agent Execution Contexts: Open-World Security Evaluation on OpenClaw11 May 2026
- 2026PRISM: Generation-Time Detection and Mitigation of Secret Leakage in Multi-Agent LLM Pipelines11 May 2026
- 2026Adaptive Defense Orchestration for RAG: A Sentinel-Strategist Architecture against Multi-Vector Attacks22 Apr 2026
- 2026MemPot: Defending Against Memory Extraction Attack with Optimized Honeypots7 Feb 2026
- 2025Black-Box Guardrail Reverse-engineering Attack6 Nov 2025
- 2025Quantifying Conversation Drift in MCP via Latent Polytope8 Aug 2025
- 2025Provably Secure Retrieval-Augmented Generation1 Aug 2025
- 2025Safeguarding Multimodal Knowledge Copyright in the RAG-as-a-Service Environment10 Jun 2025
- 2025Silent Leaks: Implicit Knowledge Extraction Attack on RAG Systems through Benign Queries21 May 2025
- 2025Unveiling the Landscape of LLM Deployment in the Wild: An Empirical Study5 May 2025
Similar, not cited
Close in wording, no citation either way
- 2025RAG Security and Privacy: Formalizing the Threat Model and Attack Surface
- 2025Adversarial Threat Vectors and Risk Mitigation for Retrieval-Augmented Generation Systems
- 2024Enhancing LLM Factual Accuracy with RAG to Counter Hallucinations: A Case Study on Domain-Specific Queries in Private Knowledge-Bases
- 2024Pirates of the RAG: Adaptively Attacking LLMs to Leak Knowledge Bases
- 2026Securing Retrieval-Augmented Generation: A Taxonomy of Attacks, Defenses, and Future Directions
- 2024Towards More Robust Retrieval-Augmented Generation: Evaluating RAG Under Adversarial Poisoning Attacks
- 2025Exploiting Web Search Tools of AI Agents for Data Exfiltration
- 2026SoK: Privacy Risks and Mitigations in Retrieval-Augmented Generation Systems