mlsanityData to 7 Oct 2026

Pandora: Jailbreak GPTs by Retrieval Augmented Generation Poisoning

arXiv 2402.08416 ↗First posted 13 Feb 2024cs.CRcited by 62

On the map: Retrieval-augmented generation → Privacy and security of RAG

also in 1 more
Abstract (arXiv)

Large Language Models~(LLMs) have gained immense popularity and are being increasingly applied in various domains. Consequently, ensuring the security of these models is of paramount importance. Jailbreak attacks, which manipulate LLMs to generate malicious content, are recognized as a significant vulnerability. While existing research has predominantly focused on direct jailbreak attacks on LLMs, there has been limited exploration of indirect methods. The integration of various plugins into LLMs, notably Retrieval Augmented Generation~(RAG), which enables LLMs to incorporate external knowledge bases into their response generation such as GPTs, introduces new avenues for indirect jailbreak attacks. To fill this gap, we investigate indirect jailbreak attacks on LLMs, particularly GPTs, introducing a novel attack vector named Retrieval Augmented Generation Poisoning. This method, Pandora, exploits the synergy between LLMs and RAG through prompt manipulation to generate unexpected responses. Pandora uses maliciously crafted content to influence the RAG process, effectively initiating jailbreak attacks. Our preliminary tests show that Pandora successfully conducts jailbreak attacks in four different scenarios, achieving higher success rates than direct attacks, with 64.3% for GPT-3.5 and 34.8% for GPT-4.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Placed in Privacy and security of RAG, one of 15 directions in Retrieval-augmented generation. Builds on 10 of its 20 references that are in the map. First posted on arXiv 13 Feb 2024.

Builds on

10 of 20 references.

References · 10

  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  2. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  3. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  4. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  5. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  6. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  7. 2023Role-Play with Large Language Modelscited by 290
  8. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 152
  9. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Modelscited by 128
  10. 2022Why So Toxic? Measuring and Triggering Toxic Behavior in Open-Domain Chatbotscited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 25; 2025: 25; 2026: 12.

62 papers.

Sort:

By area:

  1. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  2. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  3. 2024Retrieval-Augmented Generation with Graphs (GraphRAG)cited by 170
  4. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  5. 2024Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Cluescited by 57
  6. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 54
  7. 2025Practical Poisoning Attacks against Retrieval-Augmented Generationcited by 35
  8. 2024Crabs: Consuming Resource via Auto-generation for LLM-DoS Attack under Black-box Settingscited by 33
  9. 2025Towards Robust and Secure Embodied AI: A Survey on Vulnerabilities and Attackscited by 32
  10. 2025Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Surveycited by 29
  11. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 28
  12. 2024Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detectioncited by 27
38 more
  1. 2024Generating Is Believing: Membership Inference Attacks against Retrieval-Augmented Generationcited by 25
  2. 2025GraphRAG under Firecited by 23
  3. 2025RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Modelscited by 23
  4. 2024From LLMs to MLLMs: Exploring the Landscape of Multimodal Jailbreakingcited by 22
  5. 2024ConfusedPilot: Confused Deputy Risks in RAG-based LLMscited by 22
  6. 2024Towards Knowledge Checking in Retrieval-augmented Generation: A Representation Perspectivecited by 20
  7. 2024Uncovering Logit Suppression Vulnerabilities in LLM Safety Alignmentcited by 16
  8. 2024On the Vulnerability of Applying Retrieval-Augmented Generation within Knowledge-Intensive Application Domainscited by 14
  9. 2024Towards More Robust Retrieval-Augmented Generation: Evaluating RAG Under Adversarial Poisoning Attackscited by 14
  10. 2026Security Threat Modeling for Emerging AI-Agent Protocols: A Comparative Analysis of MCP, A2A, Agora, and ANPcited by 14
  11. 2025Who Taught the Lie? Responsibility Attribution for Poisoned Knowledge in Retrieval-Augmented Generationcited by 12
  12. 2025Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Modelscited by 11
  13. 2024The Dark Side of Trust: Authority Citation-Driven Jailbreak Attacks on Large Language Modelscited by 9
  14. 2024Blockchain for Large Language Model Security and Safety: A Holistic Surveycited by 8
  15. 2025Bias Amplification in RAG: Poisoning Knowledge Retrieval to Steer LLMscited by 8
  16. 2025The Hidden Threat in Plain Text: Attacking RAG Data Loaderscited by 8
  17. 2025Privacy-Preserving Federated Embedding Learning for Localized Retrieval-Augmented Generationcited by 7
  18. 2025NeuroGenPoisoning: Neuron-Guided Attacks on Retrieval-Augmented Generation of LLM via Genetic Optimization of External Knowledgecited by 7
  19. 2026SoK: Privacy Risks and Mitigations in Retrieval-Augmented Generation Systemscited by 7
  20. 2025Towards Context-Robust LLMs: A Gated Representation Fine-tuning Approachcited by 6
  21. 2024Systematic Categorization, Construction and Evaluation of New Attacks against Multi-modal Mobile GUI Agentscited by 5
  22. 2025DP-Fusion: Token-Level Differentially Private Inference for Large Language Modelscited by 5
  23. 2025CoBia: Constructed Conversations Can Trigger Otherwise Concealed Societal Biases in LLMscited by 5
  24. 2024MeTMaP: Metamorphic Testing for Detecting False Vector Matching Problems in LLM Augmented Generationcited by 4
  25. 2024Single Character Perturbations Break LLM Alignmentcited by 4
  26. 2024Model-Editing-Based Jailbreak against Safety-aligned Large Language Modelscited by 4
  27. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 4
  28. 2024GlitchProber: Advancing Effective Detection and Mitigation of Glitch Tokens in Large Language Modelscited by 3
  29. 2025Securing RAG: A Risk Assessment and Mitigation Frameworkcited by 3
  30. 2025A Few Words Can Distort Graphs: Knowledge Poisoning Attacks on Graph-based Retrieval-Augmented Generation of Large Language Modelscited by 3
  31. 2026Stateless Yet Not Forgetful: Implicit Memory as a Hidden Channel in LLMscited by 3
  32. 2026Towards Secure Retrieval-Augmented Generation: A Comprehensive Review of Threats, Defenses and Benchmarkscited by 3
  33. 2024Continuous Embedding Attacks via Clipped Inputs in Jailbreaking Large Language Modelscited by 2
  34. 2025Safety Degradation in AI Agentscited by 2
  35. 2024RAM2C: A Liberal Arts Educational Chatbot based on Retrieval-augmented Multi-role Multi-expert Collaborationcited by 1
  36. 2024Inner-Probe: Discovering Copyright-related Data Generation in LLM Architecturecited by 1
  37. 2025AC-LoRA: (Almost) Training-Free Access Control-Aware Multi-Modal LLMscited by 1
  38. 2025PD3F: A Pluggable and Dynamic DoS-Defense Framework Against Resource Consumption Attacks Targeting Large Language Modelscited by 1

50 most cited of 62 All 62 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Poisoned LangChain: Jailbreak LLMs by LangChain
  2. 2024Divide and Conquer: A Hybrid Strategy Defeats Multimodal Large Language Models
  3. 2024Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attacks
  4. 2025Retrieval-Augmented Defense: Adaptive and Controllable Jailbreak Prevention for Large Language Models
  5. 2025Scalable Defense against In-the-wild Jailbreaking Attacks with Safety Context Retrieval
  6. 2024SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Manner
  7. 2025Hoist with His Own Petard: Inducing Guardrails to Facilitate Denial-of-Service Attacks on Retrieval-Augmented Generation of LLMs
  8. 2024Stealthy Jailbreak Attacks on Large Language Models via Benign Data Mirroring