# PR-Attack: Coordinated Prompt-RAG Attacks on Retrieval-Augmented Generation in Large Language Models via Bilevel Optimization

arXiv: https://arxiv.org/abs/2504.07717 · HTML: https://mlsanity.com/paper/2504.07717/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in Privacy and security of RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 61 of its 101 references that are in the map.
- First posted on arXiv 10 Apr 2025.

## Placed in

- Retrieval-augmented generation / [Privacy and security of RAG](https://mlsanity.com/rag/privacy-and-security-of-rag/)

## Builds on (most cited first, 30 of 61)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022)
- [Bilevel Programming for Hyperparameter Optimization and Meta-Learning](https://mlsanity.com/paper/1806.04910/) (2018)
- [Weight Poisoning Attacks on Pre-trained Models](https://mlsanity.com/paper/2004.06660/) (2020)
- [Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery](https://mlsanity.com/paper/2302.03668/) (2023)
- [Visual Adversarial Examples Jailbreak Aligned Large Language Models](https://mlsanity.com/paper/2306.13213/) (2023)
- [Benchmarking Large Language Models in Retrieval-Augmented Generation](https://mlsanity.com/paper/2309.01431/) (2023)
- [Poisoning Web-Scale Training Datasets is Practical](https://mlsanity.com/paper/2302.10149/) (2023)
- [Poisoning Language Models During Instruction Tuning](https://mlsanity.com/paper/2305.00944/) (2023)
- [Bilevel Optimization: Convergence Analysis and Enhanced Design](https://mlsanity.com/paper/2010.07962/) (2020)
- [Security and Privacy Challenges of Large Language Models: A Survey](https://mlsanity.com/paper/2402.00888/) (2024)
- [Poisoning Retrieval Corpora by Injecting Adversarial Passages](https://mlsanity.com/paper/2310.19156/) (2023)
- [On the Risk of Misinformation Pollution with Large Language Models](https://mlsanity.com/paper/2305.13661/) (2023)
- [Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark](https://mlsanity.com/paper/2402.11592/) (2024)
- [Provably Faster Algorithms for Bilevel Optimization](https://mlsanity.com/paper/2106.04692/) (2021)
- [A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Models](https://mlsanity.com/paper/2402.13457/) (2024)
- [BOME! Bilevel Optimization Made Easy: A Simple First-Order Approach](https://mlsanity.com/paper/2209.08709/) (2022)
- [Optimization Methods for Personalizing Large Language Models through Retrieval Augmentation](https://mlsanity.com/paper/2404.05970/) (2024)
- [Towards Gradient-based Bilevel Optimization with Non-convex Followers and Beyond](https://mlsanity.com/paper/2110.00455/) (2021)
- [BadPrompt: Backdoor Attacks on Continuous Prompts](https://mlsanity.com/paper/2211.14719/) (2022)
- [Instruction Backdoor Attacks Against Customized LLMs](https://mlsanity.com/paper/2402.09179/) (2024)
- [A Value-Function-based Interior-point Method for Non-convex Bi-level Optimization](https://mlsanity.com/paper/2106.07991/) (2021)
- [Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection](https://mlsanity.com/paper/2308.10819/) (2023)
- [Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledge](https://mlsanity.com/paper/2403.01432/) (2024)
- [Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too?](https://mlsanity.com/paper/2212.10539/) (2022)

## Built on by (most cited first, 30 of 32)

- [Benchmarking Poisoning Attacks against Retrieval-Augmented Generation](https://mlsanity.com/paper/2505.18543/) (2025)
- [ObliInjection: Order-Oblivious Prompt Injection Attack to LLM Agents with Multi-source Data](https://mlsanity.com/paper/2512.09321/) (2025)
- [SeCon-RAG: A Two-Stage Semantic Filtering and Conflict-Free Framework for Trustworthy RAG](https://mlsanity.com/paper/2510.09710/) (2025)
- [SoK: Privacy Risks and Mitigations in Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2601.03979/) (2026)
- [RIPRAG: Hack a Black-box Retrieval-Augmented Generation Question-Answering System with Reinforcement Learning](https://mlsanity.com/paper/2510.10008/) (2025)
- [Confundo: Learning to Generate Robust Poison for Practical RAG Systems](https://mlsanity.com/paper/2602.06616/) (2026)
- [Stable Preference Optimization: A Bilevel Approach to Catastrophic Preference Shift](https://mlsanity.com/paper/2507.07723/) (2025)
- [Medusa: Cross-Modal Transferable Adversarial Attacks on Multimodal Medical Retrieval-Augmented Generation](https://mlsanity.com/paper/2511.19257/) (2025)
- [Securing Retrieval-Augmented Generation: A Taxonomy of Attacks, Defenses, and Future Directions](https://mlsanity.com/paper/2604.08304/) (2026)
- [One Single Hub Text Breaks CLIP: Identifying Vulnerabilities in Cross-Modal Encoders via Hubness](https://mlsanity.com/paper/2604.27674/) (2026)
- [Injecting Falsehoods: Adversarial Man-in-the-Middle Attacks Undermining Factual Recall in LLMs](https://mlsanity.com/paper/2511.05919/) (2025)
- [KEPo: Knowledge Evolution Poison on Graph-based Retrieval-Augmented Generation](https://mlsanity.com/paper/2603.11501/) (2026)
- [PIDP-Attack: Combining Prompt Injection with Database Poisoning Attacks on Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2603.25164/) (2026)
- [Needle-in-RAG: Prompt-Conditioned Character-Level Traceback of Poisoned Spans in Retrieved Evidence](https://mlsanity.com/paper/2605.01782/) (2026)
- [RADAR: Defending RAG Dynamically against Retrieval Corruption](https://mlsanity.com/paper/2605.22041/) (2026)
- [SilentRetrieval: Hijacking Retrieval-Augmented Generation via Semantically-Preserving Adversarial Data Poisoning](https://mlsanity.com/paper/2605.28074/) (2026)
- [DiscourseFlip: An Oblique Discourse-Level Opinion Manipulation Attack against Black-box Retrieval-Augmented Generation](https://mlsanity.com/paper/2606.01212/) (2026)
- [PRA-RAG: Provably Robust Aggregation in Retrieval-Augmented Generation against Retrieval Corruption](https://mlsanity.com/paper/2607.00012/) (2026)
- [Knowledge Base Poisoning Attacks and Defense for Policy-Aware LLM-RAG Framework](https://mlsanity.com/paper/2607.04379/) (2026)
- [Retrieved But Not Reliable: A Survey on Attacks, and Defenses in Retrieval-Augmented Generation](https://mlsanity.com/paper/2608.24977/) (2026)
- [From Language to Logic: A Bi-Level Framework for Structured Reasoning](https://mlsanity.com/paper/2507.08501/) (2025)
- [LLM-Driven Multi-Turn Task-Oriented Dialogue Synthesis for Realistic Reasoning](https://mlsanity.com/paper/2602.23610/) (2026)
- [BiRD: A Bidirectional Ranking Defense Mechanism for Retrieval Augmented Generation](https://mlsanity.com/paper/2605.20123/) (2026)
- [FD-RAG: Federated Dual-System Retrieval-Augmented Generation](https://mlsanity.com/paper/2605.27432/) (2026)
- [CodeSentinel: A Three-Layer Defense Against Indirect Prompt Injection in Code Contexts](https://mlsanity.com/paper/2606.19235/) (2026)
- [First-order Constrained Trilevel Optimization Over Distributed Networks for Robust Coreset Selection](https://mlsanity.com/paper/2607.27632/) (2026)
- [PURPOSE: Poisoning Conflict Resolution in RAG via Proxy-Fact-Grounded Updates](https://mlsanity.com/paper/2608.04756/) (2026)
- [Lazy Grounding: Attacking Search Agents with Factual Evidence](https://mlsanity.com/paper/2608.30303/) (2026)
- [ViTeGate: Visual-Textual Triggered Knowledge Poisoning for Vision-Language Retrieval-Augmented Generation](https://mlsanity.com/paper/2609.14685/) (2026)
- [Detecting and Localizing Segment-Level Poisoning in Multi-Source LLM-Agent Inputs](https://mlsanity.com/paper/2609.14723/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable performance across a wide range of applications, e.g., medical question-answering, mathematical sciences, and code generation. However, they also exhibit inherent limitations, such as outdated knowledge and susceptibility to hallucinations. Retrieval-Augmented Generation (RAG) has emerged as a promising paradigm to address these issues, but it also introduces new vulnerabilities. Recent efforts have focused on the security of RAG-based LLMs, yet existing attack methods face three critical challenges: (1) their effectiveness declines sharply when only a limited number of poisoned texts can be injected into the knowledge database, (2) they lack sufficient stealth, as the attacks are often detectable by anomaly detection systems, which compromises their effectiveness, and (3) they rely on heuristic approaches to generate poisoned texts, lacking formal optimization frameworks and theoretic guarantees, which limits their effectiveness and applicability. To address these issues, we propose coordinated Prompt-RAG attack (PR-attack), a novel optimization-driven attack that introduces a small number of poisoned texts into the knowledge database while embedding a backdoor trigger within the prompt. When activated, the trigger causes the LLM to generate pre-designed responses to targeted queries, while maintaining normal behavior in other contexts. This ensures both high effectiveness and stealth. We formulate the attack generation process as a bilevel optimization problem leveraging a principled optimization framework to develop optimal poisoned texts and triggers. Extensive experiments across diverse LLMs and datasets demonstrate the effectiveness of PR-Attack, achieving a high attack success rate even with a limited number of poisoned texts and significantly improved stealth compared to existing methods.