# Human-Imperceptible Retrieval Poisoning Attacks in LLM-Powered Applications

arXiv: https://arxiv.org/abs/2404.17196 · HTML: https://mlsanity.com/paper/2404.17196/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Placed in Privacy and security of RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 13 of its 24 references that are in the map.
- First posted on arXiv 26 Apr 2024.

## Placed in

- Retrieval-augmented generation / [Privacy and security of RAG](https://mlsanity.com/rag/privacy-and-security-of-rag/)

## Builds on (most cited first, 13 of 13)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [Prompt Injection attack against LLM-integrated Applications](https://mlsanity.com/paper/2306.05499/) (2023)
- [Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation](https://mlsanity.com/paper/2310.06987/) (2023)
- [Neural Machine Translation: A Review and Survey](https://mlsanity.com/paper/1912.02047/) (2019)
- [Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training](https://mlsanity.com/paper/2110.14883/) (2021)
- [Prompting Frameworks for Large Language Models: A Survey](https://mlsanity.com/paper/2311.12785/) (2023)
- [On the Transferability of Adversarial Attacksagainst Neural Text Classifier](https://mlsanity.com/paper/2011.08558/) (2020)

## Built on by (most cited first, 30 of 33)

- [Safety at Scale: A Comprehensive Survey of Large Model and Agent Safety](https://mlsanity.com/paper/2502.05206/) (2025)
- [From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflows](https://mlsanity.com/paper/2506.23260/) (2025)
- [PR-Attack: Coordinated Prompt-RAG Attacks on Retrieval-Augmented Generation in Large Language Models via Bilevel Optimization](https://mlsanity.com/paper/2504.07717/) (2025)
- [Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges](https://mlsanity.com/paper/2510.23883/) (2025)
- [FlippedRAG: Black-Box Opinion Manipulation Adversarial Attacks to Retrieval-Augmented Generation Models](https://mlsanity.com/paper/2501.02968/) (2025)
- [Adversarial Decoding: Generating Readable Documents for Adversarial Objectives](https://mlsanity.com/paper/2410.02163/) (2024)
- [SafeEmbodAI: a Safety Framework for Mobile Robots in Embodied AI Systems](https://mlsanity.com/paper/2409.01630/) (2024)
- [Prompt Injection 2.0: Hybrid AI Threats](https://mlsanity.com/paper/2507.13169/) (2025)
- [Poisoned-MRAG: Knowledge Poisoning Attacks to Multimodal Retrieval Augmented Generation](https://mlsanity.com/paper/2503.06254/) (2025)
- [Black-Box Opinion Manipulation Attacks to Retrieval-Augmented Generation of Large Language Models](https://mlsanity.com/paper/2407.13757/) (2024)
- [The Dark Side of Trust: Authority Citation-Driven Jailbreak Attacks on Large Language Models](https://mlsanity.com/paper/2411.11407/) (2024)
- [SoK: Privacy Risks and Mitigations in Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2601.03979/) (2026)
- [Exploring Backdoor Attack and Defense for LLM-empowered Recommendations](https://mlsanity.com/paper/2504.11182/) (2025)
- [POISONCRAFT: Practical Poisoning of Retrieval-Augmented Generation for Large Language Models](https://mlsanity.com/paper/2505.06579/) (2025)
- [Understanding the Anchoring Effect of LLM with Synthetic Data: Existence, Mechanism, and Potential Mitigations](https://mlsanity.com/paper/2505.15392/) (2025)
- [Towards Advancing Code Generation with Large Language Models: A Research Roadmap](https://mlsanity.com/paper/2501.11354/) (2025)
- [Uncovering Competing Poisoning Attacks in Retrieval-Augmented Generation](https://mlsanity.com/paper/2505.12574/) (2025)
- [A First Look at the Security Issues in the Model Context Protocol Ecosystem](https://mlsanity.com/paper/2510.16558/) (2025)
- [Evaluating Privilege Usage of Agents with Real-World Tools](https://mlsanity.com/paper/2603.28166/) (2026)
- [The Best Defense is a Good Offense: Countering LLM-Powered Cyberattacks](https://mlsanity.com/paper/2410.15396/) (2024)
- [OMNISEC: LLM-Driven Provenance-based Intrusion Detection via Retrieval-Augmented Behavior Prompting](https://mlsanity.com/paper/2503.03108/) (2025)
- [Architecting Trust in Artificial Epistemic Agents](https://mlsanity.com/paper/2603.02960/) (2026)
- [Prompt Overflow: What the Guardrail Inspects Is Not What the Model Infers](https://mlsanity.com/paper/2605.23196/) (2026)
- [On the Validity of Traditional Vulnerability Scoring Systems for Adversarial Attacks against LLMs](https://mlsanity.com/paper/2412.20087/) (2024)
- [Medusa: Cross-Modal Transferable Adversarial Attacks on Multimodal Medical Retrieval-Augmented Generation](https://mlsanity.com/paper/2511.19257/) (2025)
- [Securing Retrieval-Augmented Generation: A Taxonomy of Attacks, Defenses, and Future Directions](https://mlsanity.com/paper/2604.08304/) (2026)
- [Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation](https://mlsanity.com/paper/2508.10404/) (2025)
- [A Survey: Towards Privacy and Security in Mobile Large Language Models](https://mlsanity.com/paper/2509.02411/) (2025)
- [LLM-Oriented Information Retrieval: A Denoising-First Perspective](https://mlsanity.com/paper/2605.00505/) (2026)
- [Needle-in-RAG: Prompt-Conditioned Character-Level Traceback of Poisoned Spans in Retrieved Evidence](https://mlsanity.com/paper/2605.01782/) (2026)

## Abstract (arXiv)

Presently, with the assistance of advanced LLM application development frameworks, more and more LLM-powered applications can effortlessly augment the LLMs' knowledge with external content using the retrieval augmented generation (RAG) technique. However, these frameworks' designs do not have sufficient consideration of the risk of external content, thereby allowing attackers to undermine the applications developed with these frameworks. In this paper, we reveal a new threat to LLM-powered applications, termed retrieval poisoning, where attackers can guide the application to yield malicious responses during the RAG process. Specifically, through the analysis of LLM application frameworks, attackers can craft documents visually indistinguishable from benign ones. Despite the documents providing correct information, once they are used as reference sources for RAG, the application is misled into generating incorrect responses. Our preliminary experiments indicate that attackers can mislead LLMs with an 88.33% success rate, and achieve a 66.67% success rate in the real-world application, demonstrating the potential impact of retrieval poisoning.