Eyes-on-Me: Scalable RAG Poisoning through Transferable Attention-Steering Attractors
On the map: Retrieval-augmented generation → Privacy and security of RAG
also in 1 more
Abstract (arXiv)
Existing data poisoning attacks on retrieval-augmented generation (RAG) systems scale poorly because they require costly optimization of poisoned documents for each target phrase. We introduce Eyes-on-Me, a modular attack that decomposes an adversarial document into reusable **Attention Attractors** and **Focus Regions**. Attractors are optimized to direct attention to the Focus Region. Attackers can then insert semantic baits for the retriever or malicious instructions for the generator, adapting to new targets at near zero cost. This is achieved by steering a small subset of attention heads that we empirically identify as strongly correlated with attack success. Across 18 end-to-end RAG settings (3 datasets × 2 retrievers × 3 generators), Eyes-on-Me raises average attack success rates from 21.9 to 57.8 (+35.9 points, 2.6× over prior work). A single optimized attractor transfers to unseen black box retrievers and generators without retraining. Our findings establish a scalable paradigm for RAG data poisoning and show that modular, reusable components pose a practical threat to modern AI systems. They also contribute to interpretability research by revealing a strong link between attention concentration and model outputs.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Privacy and security of RAG, one of 15 directions in Retrieval-augmented generation. Builds on 36 of its 59 references that are in the map. First posted on arXiv 1 Oct 2025.
Builds on
36 of 59 references.
Closest ancestors · 4
Same direction, newest first
- 2025Practical Poisoning Attacks against Retrieval-Augmented Generation4 Apr
- 2024HijackRAG: Hijacking Attacks against Retrieval-Augmented Large Language Models30 Oct
- 2024TrojanRAG: Retrieval-Augmented Generation Can Be Backdoor Driver in Large Language Models22 May
- 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models12 Feb
Other references · 32
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
- 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
- 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
- 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
- 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
- 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
- 2022In-context Learning and Induction Headscited by 930
- 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
- 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
- 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
- 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
20 more
- 2023Detecting Language Model Attacks with Perplexitycited by 411
- 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 402
- 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
- 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMscited by 185
- 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 160
- 2023Poisoning Retrieval Corpora by Injecting Adversarial Passagescited by 153
- 2023Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMscited by 82
- 2024Attention Tracker: Detecting Prompt Injection Attacks in LLMscited by 82
- 2023JailGuard: A Universal Detection Framework for LLM Prompt-based Attackscited by 46
- 2024AttnGCG: Enhancing Jailbreaking Attacks on LLMs with Attention Manipulationcited by 31
- 2024Faster-GCG: Efficient Discrete Optimization Jailbreak Attacks against Aligned Large Language Modelscited by 23
- 2024Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Modelscited by 23
- 2024RL-JACK: Reinforcement Learning-powered Black-box Jailbreaking Attack against LLMscited by 21
- 2025Spotlight Your Instructions: Instruction-following with Dynamic Attention Steeringcited by 18
- 2024Backdoor Attacks on Dense Retrieval via Public and Unintentional Triggerscited by 13
- 2025xJailbreak: Representation Space Guided Reinforcement Learning for Interpretable LLM Jailbreakingcited by 10
- 2024LLMSteer: Improving Long-Context LLM Inference by Steering Attention on Reused Contextscited by 7
- 2024Attention Instruction: Amplifying Attention in the Middle via Promptingcited by 6
- 2025Attention Eclipse: Manipulating Attention to Bypass LLM Safety-Alignmentcited by 6
- 2025Jailbreak Transferability Emerges from Shared Representationscited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026When Context Bites: Detecting RAG Poisoning via Document-Level Attention Collapse
- 2026Confundo: Learning to Generate Robust Poison for Practical RAG Systems
- 2026Semantic Chameleon: Corpus-Dependent Poisoning Attacks and Defenses in RAG Systems
- 2025CPA-RAG:Covert Poisoning Attacks on Retrieval-Augmented Generation in Large Language Models
- 2026Influence Factors on RAG Poisoning
- 2025Through the Stealth Lens: Attention-Aware Defenses Against Poisoning in RAG
- 2025POISONCRAFT: Practical Poisoning of Retrieval-Augmented Generation for Large Language Models
- 2025MM-PoisonRAG: Disrupting Multimodal RAG with Local and Global Poisoning Attacks