mlsanityData to 7 Oct 2026

RevPRAG: Revealing Poisoning Attacks in Retrieval-Augmented Generation through LLM Activation Analysis

arXiv 2411.18948 ↗First posted 28 Nov 2024cs.CR cs.AIcited by 33

On the map: Retrieval-augmented generation → Privacy and security of RAG

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) enriches the input to LLMs by retrieving information from the relevant knowledge database, enabling them to produce responses that are more accurate and contextually appropriate. It is worth noting that the knowledge database, being sourced from publicly available channels such as Wikipedia, inevitably introduces a new attack surface. RAG poisoning involves injecting malicious texts into the knowledge database, ultimately leading to the generation of the attacker's target response (also called poisoned response). However, there are currently limited methods available for detecting such poisoning attacks. We aim to bridge the gap in this work. Particularly, we introduce RevPRAG, a flexible and automated detection pipeline that leverages the activations of LLMs for poisoned response detection. Our investigation uncovers distinct patterns in LLMs' activations when generating correct responses versus poisoned responses. Our results on multiple benchmark datasets and RAG architectures show our approach could achieve 98% true positive rate, while maintaining false positive rates close to 1%.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in Privacy and security of RAG, one of 15 directions in Retrieval-augmented generation. Builds on 26 of its 49 references that are in the map. First posted on arXiv 28 Nov 2024.

Builds on

26 of 49 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Certifiably Robust RAG against Retrieval Corruption24 May
  2. 2024Typos that Broke the RAG's Back: Genetic Attack on RAG Pipeline by Simulating Documents in the Wild via Low-level Perturbations22 Apr

Other references · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  5. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  6. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  7. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  8. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  9. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  10. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  11. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  12. 2022Mass-Editing Memory in a Transformercited by 976
12 more
  1. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  2. 2021Generalized Out-of-Distribution Detection: A Surveycited by 796
  3. 2020On Adaptive Attacks to Adversarial Example Defensescited by 547
  4. 2017Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methodscited by 542
  5. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 437
  6. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  7. 2023Poisoning Retrieval Corpora by Injecting Adversarial Passagescited by 153
  8. 2023Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noisecited by 52
  9. 2023Making LLMs Worth Every Penny: Resource-Limited Text Classification in Bankingcited by 36
  10. 2021Evading Adversarial Example Detection Defenses with Orthogonal Projected Gradient Descentcited by 34
  11. 2023Defending Pre-trained Language Models as Few-shot Learners against Backdoor Attackscited by 32
  12. 2023A LLM Assisted Exploitation of AI-Guardiancited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 27.

33 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2025MemoryGraft: Persistent Compromise of LLM Agents via Poisoned Experience Retrievalcited by 65
  3. 2026A Survey on Long-Term Memory Security in LLM Agents: Attacks, Defenses, and Governance Across the Memory Lifecyclecited by 21
  4. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  5. 2025RIPRAG: Hack a Black-box Retrieval-Augmented Generation Question-Answering System with Reinforcement Learningcited by 5
  6. 2026Towards Security-Auditable LLM Agents: A Unified Graph Representationcited by 4
  7. 2026Addressing Corpus Knowledge Poisoning Attacks on RAG Using Sparse Attentioncited by 3
  8. 2026MEMSAD: Gradient-Coupled Anomaly Detection for Memory Poisoning in Retrieval-Augmented Agentscited by 3
  9. 2026MemPoison: Uncovering Persistent Memory Threats and Structural Blind Spots in LLM Agentscited by 3
  10. 2025FuncPoison: Poisoning Function Library to Hijack Multi-agent Autonomous Driving Systemscited by 2
  11. 2026Adversarial Intent is a Latent Variable: Stateful Trust Inference for Securing Multimodal Agentic RAGcited by 2
  12. 2026ProGRank: Probe-Gradient Reranking to Defend Dense-Retriever RAG from Corpus Poisoningcited by 2
21 more
  1. 2026Securing Retrieval-Augmented Generation: A Taxonomy of Attacks, Defenses, and Future Directionscited by 2
  2. 2026One Single Hub Text Breaks CLIP: Identifying Vulnerabilities in Cross-Modal Encoders via Hubnesscited by 2
  3. 2026Retrieval Pivot Attacks in Hybrid RAG: Measuring and Mitigating Amplified Leakage from Vector Seeds to Graph Expansioncited by 1
  4. 2026Needle-in-RAG: Prompt-Conditioned Character-Level Traceback of Poisoned Spans in Retrieved Evidencecited by 1
  5. 2026Cordon-MAS: Defending RAG against Knowledge Poisoning via Information-Flow Controlcited by 1
  6. 2026When Agents Go Rogue: Activation-Based Detection of Malicious Behaviors in Multi-Agent Systemscited by 1
  7. 2026When Context Bites: Detecting RAG Poisoning via Document-Level Attention Collapsecited by 1
  8. 2025Mitigating GenAI-Powered Evidence Pollution for Out-Of-Context Misinformation Detectioncited by 0
  9. 2026ToolFlood: Beyond Selection -- Hiding Valid Tools from LLM Agents via Semantic Coveringcited by 0
  10. 2026Towards Unsupervised Adversarial Document Detection in Retrieval Augmented Generation Systemscited by 0
  11. 2026When Poison Fails After Retrieval: Revisiting Corpus Poisoning under Chunking and Reranking Pipelinescited by 0
  12. 2026Tracing Target Answers in Poisoned Retrieval Corpora via Token Influence Attributioncited by 0
  13. 2026Large Language Models in Misinformation Ecosystems: Misuse, Defense, and Vulnerabilitycited by 0
  14. 2026TriShieldRAG: 3 Rings, One Blind Spot in Layered Defenses for Retrieval-Augmented Generationcited by 0
  15. 2026RAGSieve: Self-Referenced Local Contrast for Knowledge-Poison Detection in Retrieval-Augmented Generationcited by 0
  16. 2026Retrieved But Not Reliable: A Survey on Attacks, and Defenses in Retrieval-Augmented Generationcited by 0
  17. 2026CIG-MIA: Context-Induced Information Gain Membership Inference Attacks against Retrieval-Augmented Generationcited by 0
  18. 2026Detecting and Localizing Segment-Level Poisoning in Multi-Source LLM-Agent Inputscited by 0
  19. 2026Toward Secure AI-Powered Penetration Testing Agents: Security Threats, Guardrails, and Architectural Perspectivescited by 0
  20. 2026InceptionRAG: Stealthy Poisoning Attack Against Retrieval-Augmented Generationcited by 0
  21. 2026Market Signal Injection: Adversarial Context Manipulation of LLM Pricing Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Traceback of Poisoning Attacks to Retrieval-Augmented Generation
  2. 2026Confundo: Learning to Generate Robust Poison for Practical RAG Systems
  3. 2025Practical Poisoning Attacks against Retrieval-Augmented Generation
  4. 2025RAG Safety: Exploring Knowledge Poisoning Attacks to Retrieval-Augmented Generation
  5. 2025Secure Retrieval-Augmented Generation against Poisoning Attacks
  6. 2024Human-Imperceptible Retrieval Poisoning Attacks in LLM-Powered Applications
  7. 2026Influence Factors on RAG Poisoning
  8. 2025Benchmarking Poisoning Attacks against Retrieval-Augmented Generation