mlsanityData to 7 Oct 2026

Prompt Perturbation in Retrieval-Augmented Generation based Large Language Models

arXiv 2402.07179 ↗First posted 11 Feb 2024cs.CL cs.IRcited by 17

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

The robustness of large language models (LLMs) becomes increasingly important as their use rapidly grows in a wide range of domains. Retrieval-Augmented Generation (RAG) is considered as a means to improve the trustworthiness of text generation from LLMs. However, how the outputs from RAG-based LLMs are affected by slightly different inputs is not well studied. In this work, we find that the insertion of even a short prefix to the prompt leads to the generation of outputs far away from factually correct answers. We systematically evaluate the effect of such prefixes on RAG by introducing a novel optimization technique called Gradient Guided Prompt Perturbation (GGPP). GGPP achieves a high success rate in steering outputs of RAG-based LLMs to targeted wrong answers. It can also cope with instructions in the prompts requesting to ignore irrelevant context. We also exploit LLMs' neuron activation difference between prompts with and without GGPP perturbations to give a method that improves the robustness of RAG-based LLMs through a highly effective detector trained on neuron activation triggered by GGPP generated prompts. Our evaluation on open-sourced LLMs demonstrates the effectiveness of our methods.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 31 of its 50 references that are in the map. First posted on arXiv 11 Feb 2024.

Builds on

31 of 50 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  5. 2023Qwen Technical Reportcited by 3,273
  6. 2023Mistral 7Bcited by 3,065
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  8. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  9. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  10. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  11. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 853
  12. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
19 more
  1. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  2. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  3. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  4. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  5. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  6. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  7. 2022Memory-Based Model Editing at Scalecited by 443
  8. 2023Augmented Language Models: a Surveycited by 398
  9. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
  10. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  11. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  12. 2022Predictability and Surprise in Large Generative Modelscited by 209
  13. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 208
  14. 2022Re3: Generating Longer Stories With Recursive Reprompting and Revisioncited by 193
  15. 2024RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculturecited by 80
  16. 2023Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Modelscited by 67
  17. 2023How Does Generative Retrieval Scale to Millions of Passages?cited by 47
  18. 2020TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLPcited by 45
  19. 2023Vector Search with OpenAI Embeddings: Lucene Is All You Needcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 10; 2026: 4.

17 papers.

Sort:

By area:

  1. 2024Is My Data in Your Retrieval Database? Membership Inference Attacks Against Retrieval Augmented Generationcited by 44
  2. 2025PR-Attack: Coordinated Prompt-RAG Attacks on Retrieval-Augmented Generation in Large Language Models via Bilevel Optimizationcited by 29
  3. 2024Generating Is Believing: Membership Inference Attacks against Retrieval-Augmented Generationcited by 25
  4. 2025Enhancing LLM Robustness to Perturbed Instructions: An Empirical Studycited by 17
  5. 2025Rescuing the Unpoisoned: Efficient Defense against Knowledge Corruption Attacks on RAG Systemscited by 9
  6. 2024Blockchain for Large Language Model Security and Safety: A Holistic Surveycited by 8
  7. 2025Are Robust LLM Fingerprints Adversarially Robust?cited by 7
  8. 2025AGRAG: Advanced Graph-based Retrieval-Augmented Generation for LLMscited by 7
  9. 2026SoK: Privacy Risks and Mitigations in Retrieval-Augmented Generation Systemscited by 7
  10. 2025DeRAG: Black-box Adversarial Attacks on Multiple Retrieval-Augmented Generation Applications via Prompt Injectioncited by 5
  11. 2025RIPRAG: Hack a Black-box Retrieval-Augmented Generation Question-Answering System with Reinforcement Learningcited by 5
  12. 2025Code Roulette: How Prompt Variability Affects LLM Code Generationcited by 4
5 more
  1. 2026Towards Security-Auditable LLM Agents: A Unified Graph Representationcited by 4
  2. 2025Improving Consistency in Retrieval-Augmented Systems with Group Similarity Rewardscited by 2
  3. 2026PIDP-Attack: Combining Prompt Injection with Database Poisoning Attacks on Retrieval-Augmented Generation Systemscited by 1
  4. 2025LFD: Layer Fused Decoding to Exploit External Knowledge in Retrieval-Augmented Generationcited by 0
  5. 2026Language Models as Interfaces, Not Oracles: A Hybrid LLM-ML System for Pediatric Appendicitiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Establishing Performance Baselines in Fine-Tuning, Retrieval-Augmented Generation and Soft-Prompting for Non-Specialist LLM Users
  2. 2026Evaluating Prompt Engineering Techniques for RAG in Small Language Models: A Multi-Hop QA Approach
  3. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Models
  4. 2025Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers
  5. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG
  6. 2024Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation
  7. 2025On the Diminishing Returns of Complex Robust RAG Training in the Era of Powerful LLMs
  8. 2024PA-RAG: RAG Alignment via Multi-Perspective Preference Optimization