mlsanityData to 14 Oct 2026

Through the Stealth Lens: Attention-Aware Defenses Against Poisoning in RAG

arXiv 2506.04390 ↗First posted 4 Jun 2025cs.CR cs.AIcited by 3

On the map: Retrieval-augmented generation → Privacy and security of RAG

Abstract (arXiv)

Retrieval-augmented generation (RAG) systems are vulnerable to attacks that inject poisoned passages into the retrieved context, even at low corruption rates. We show that existing attacks are not designed to be stealthy, allowing reliable detection and mitigation. We formalize a distinguishability-based security game to quantify stealth for such attacks. If a few poisoned passages control the response, they must bias the inference process more than the benign ones, inherently compromising stealth. This motivates analyzing intermediate signals of LLMs, such as attention weights, to approximate the influence of different passages on the response. Leveraging attention weights, we introduce the Normalized Passage Attention Score (NPAS) and a lightweight Attention-Variance Filter (AV Filter) that flags anomalous passages. Our method improves robustness, yielding up to ~ 20% higher accuracy than baseline defenses. We also develop adaptive attacks that attempt to conceal such anomalies, achieving up to 35% success rate and underscoring the challenges of achieving true stealth in poisoning RAG systems.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Privacy and security of RAG, one of 15 directions in Retrieval-augmented generation. Builds on 37 of its 52 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

37 of 52 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Practical Poisoning Attacks against Retrieval-Augmented Generation4 Apr
  2. 2025The RAG Paradox: A Black-Box Attack Exploiting Unintentional Vulnerabilities in Retrieval-Augmented Generation Systems28 Feb
  3. 2024Certifiably Robust RAG against Retrieval Corruption24 May

Other references · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  7. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  8. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  10. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  11. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  12. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
22 more
  1. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  2. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  3. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  4. 2020Quantifying Attention Flow in Transformerscited by 775
  5. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  6. 2023Detecting Language Model Attacks with Perplexitycited by 431
  7. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 290
  8. 2022RealTime QA: What's the Answer Right Now?cited by 244
  9. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 243
  10. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 216
  11. 2023Poisoning Retrieval Corpora by Injecting Adversarial Passagescited by 159
  12. 2023On the Risk of Misinformation Pollution with Large Language Modelscited by 128
  13. 2022Re2G: Retrieve, Rerank, Generatecited by 102
  14. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 93
  15. 2023Attention Sorting Combats Recency Bias In Long Context Language Modelscited by 90
  16. 2023WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipediacited by 67
  17. 2022Synthetic Disinformation Attacks on Automated Fact Verification Systemscited by 44
  18. 2021Attacking Open-domain Question Answering by Injecting Misinformationcited by 41
  19. 2024Serial Position Effects of Large Language Modelscited by 30
  20. 2025Collapse of Dense Retrievers: Short, Early, and Literal Biases Outranking Factual Evidencecited by 16
  21. 2021More Robust Dense Retrieval with Contrastive Dual Learningcited by 15
  22. 2022Defending Against Disinformation Attacks in Open-Domain Question Answeringcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Cordon-MAS: Defending RAG against Knowledge Poisoning via Information-Flow Controlcited by 1
  2. 2026Retrieved But Not Reliable: A Survey on Attacks, and Defenses in Retrieval-Augmented Generationcited by 1
  3. 2026MIND: Lightweight and Effective Memory Injection Defense for LLM Agents via Intent-Aware Information Bottleneckcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026When Context Bites: Detecting RAG Poisoning via Document-Level Attention Collapse
  2. 2025Eyes-on-Me: Scalable RAG Poisoning through Transferable Attention-Steering Attractors
  3. 2026Confundo: Learning to Generate Robust Poison for Practical RAG Systems
  4. 2025Benchmarking Poisoning Attacks against Retrieval-Augmented Generation
  5. 2026RouteGuard: Internal-Signal Detection of Skill Poisoning in LLM Agents
  6. 2026Reasoning Reduces the Influence of Poisoned Context in RAG
  7. 2024RevPRAG: Revealing Poisoning Attacks in Retrieval-Augmented Generation through LLM Activation Analysis
  8. 2026Detecting RAG Extraction Attack via Dual-Path Runtime Integrity Game