mlsanityData to 7 Oct 2026

ConfusedPilot: Confused Deputy Risks in RAG-based LLMs

arXiv 2408.04870 ↗First posted 9 Aug 2024cs.CR cs.AIcited by 22

On the map: Retrieval-augmented generation → Privacy and security of RAG

Abstract (arXiv)

Retrieval augmented generation (RAG) is a process where a large language model (LLM) retrieves useful information from a database and then generates the responses. It is becoming popular in enterprise settings for daily business operations. For example, Copilot for Microsoft 365 has accumulated millions of businesses. However, the security implications of adopting such RAG-based systems are unclear. In this paper, we introduce ConfusedPilot, a class of security vulnerabilities of RAG systems that confuse Copilot and cause integrity and confidentiality violations in its responses. First, we investigate a vulnerability that embeds malicious text in the modified prompt in RAG, corrupting the responses generated by the LLM. Second, we demonstrate a vulnerability that leaks secret data, which leverages the caching mechanism during retrieval. Third, we investigate how both vulnerabilities can be exploited to propagate misinformation within the enterprise and ultimately impact its operations, such as sales and manufacturing. We also discuss the root cause of these attacks by investigating the architecture of a RAG-based system. This study highlights the security vulnerabilities in today's RAG-based systems and proposes design guidelines to secure future RAG-based systems.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Privacy and security of RAG, one of 15 directions in Retrieval-augmented generation. Builds on 40 of its 68 references that are in the map. First posted on arXiv 9 Aug 2024.

Builds on

40 of 68 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024TrojanRAG: Retrieval-Augmented Generation Can Be Backdoor Driver in Large Language Models22 May
  2. 2024The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)23 Feb
  3. 2024Pandora: Jailbreak GPTs by Retrieval Augmented Generation Poisoning13 Feb

Other references · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  7. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  8. 2017Billion-scale similarity search with GPUscited by 2,477
  9. 2020Extracting Training Data from Large Language Modelscited by 2,144
  10. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  11. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  12. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
25 more
  1. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  2. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  3. 2023Extracting Training Data from Diffusion Modelscited by 723
  4. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  5. 2016Stealing Machine Learning Models via Prediction APIscited by 602
  6. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  7. 2018Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networkscited by 500
  8. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  9. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  10. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  11. 2023Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoningcited by 326
  12. 2024G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answeringcited by 217
  13. 2023Query Rewriting for Retrieval-Augmented Large Language Modelscited by 211
  14. 2024Stealing Part of a Production Language Modelcited by 152
  15. 2024Seven Failure Points When Engineering a Retrieval Augmented Generation Systemcited by 138
  16. 2022Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answeringcited by 126
  17. 2024AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attackscited by 94
  18. 2024A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systemscited by 74
  19. 2021Poisoning the Unlabeled Dataset of Semi-Supervised Learningcited by 44
  20. 2023TrojLLM: A Black-box Trojan Prompt Attack on Large Language Modelscited by 29
  21. 2023Copy Is All You Needcited by 25
  22. 2021ISEEQ: Information Seeking Question Generation using Dynamic Meta-Information Retrieval and Knowledge Graphscited by 22
  23. 2023Rethinking Privacy in Machine Learning Pipelines from an Information Flow Control Perspectivecited by 16
  24. 2023IAG: Induction-Augmented Generation Framework for Answering Reasoning Questionscited by 15
  25. 2023Information Flow Control in Machine Learning through Modular Model Architecturecited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 14.

22 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2025Multi-Agent Systems Execute Arbitrary Malicious Codecited by 52
  3. 2025GraphRAG under Firecited by 23
  4. 2024Pirates of the RAG: Adaptively Attacking LLMs to Leak Knowledge Basescited by 17
  5. 2025Benchmarking Poisoning Attacks against Retrieval-Augmented Generationcited by 17
  6. 2026A Framework for Formalizing LLM Agent Securitycited by 16
  7. 2025SeCon-RAG: A Two-Stage Semantic Filtering and Conflict-Free Framework for Trustworthy RAGcited by 12
  8. 2026SoK: Privacy Risks and Mitigations in Retrieval-Augmented Generation Systemscited by 7
  9. 2026Security Considerations for Artificial Intelligence Agentscited by 7
  10. 2026Trust Me, Import This: Dependency Steering Attacks via Malicious Agent Skillscited by 4
  11. 2026Do Coding Agents Understand Least-Privilege Authorization?cited by 4
  12. 2026SD-RAG: A Prompt-Injection-Resilient Framework for Selective Disclosure in Retrieval-Augmented Generationcited by 3
10 more
  1. 2026Securing Retrieval-Augmented Generation: A Taxonomy of Attacks, Defenses, and Future Directionscited by 2
  2. 2024SoK: A Systems Perspective on Compound AI Threats and Countermeasurescited by 1
  3. 2026From Thinker to Society: Security in Hierarchical Autonomy Evolution of AI Agentscited by 1
  4. 2026Cascade: Composing Software-Hardware Attack Gadgets for Adversarial Threat Amplification in Compound AI Systemscited by 1
  5. 2026RAGShield: Detecting Numerical Claim Manipulation in Government RAG Systemscited by 1
  6. 2026Conflict-Aware Retriever Editing for Knowledge Injection Attacks on LLM-Based RAG Systemscited by 1
  7. 2026Blockchain Empowered Trustworthy Agent Networks: Foundations, Taxonomy, and Future Directionscited by 1
  8. 2025RAGRank: Using PageRank to Counter Poisoning in CTI LLM Pipelinescited by 0
  9. 2026DenialRAG: Single-Document RAG Poisoning via Embedded Parametric Denialcited by 0
  10. 2026The Natural Language Interaction Protocol and Standard for AI Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Securing RAG: A Risk Assessment and Mitigation Framework
  2. 2025Token-Level Precise Attack on RAG: Searching for the Best Alternatives to Mislead Generation
  3. 2025RAG Security and Privacy: Formalizing the Threat Model and Attack Surface
  4. 2024Rag and Roll: An End-to-End Evaluation of Indirect Prompt Manipulations in LLM-based Application Frameworks
  5. 2026Towards Secure Retrieval-Augmented Generation: A Comprehensive Review of Threats, Defenses and Benchmarks
  6. 2026RAG-CT: Mitigating Privacy Risks on Retrieval-Augmented Generation Systems via Scanning Prompt Distribution
  7. 2025RAG Safety: Exploring Knowledge Poisoning Attacks to Retrieval-Augmented Generation
  8. 2026Security and Privacy in Retrieval-Augmented Generation: Architectures, Threats, Defenses, and Future Directions for Building Trustworthy Systems