mlsanityData to 7 Oct 2026

Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation

arXiv 2404.06910 ↗First posted 10 Apr 2024cs.CL cs.AI cs.LGcited by 15

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Despite the successes of large language models (LLMs), they exhibit significant drawbacks, particularly when processing long contexts. Their inference cost scales quadratically with respect to sequence length, making it expensive for deployment in some real-world text processing applications, such as retrieval-augmented generation (RAG). Additionally, LLMs also exhibit the "distraction phenomenon", where irrelevant context in the prompt degrades output quality. To address these drawbacks, we propose a novel RAG prompting methodology, *superposition prompting*, which can be directly applied to pre-trained transformer-based LLMs *without the need for fine-tuning*. At a high level, superposition prompting allows the LLM to process input documents in parallel *prompt paths*, discarding paths once they are deemed irrelevant. We demonstrate the capability of our method to simultaneously enhance time efficiency across a variety of question-answering benchmarks using multiple pre-trained LLMs. Furthermore, our technique significantly improves accuracy when the retrieved context is large relative the context the model was trained on. For example, our approach facilitates a 93x reduction in compute time while *improving* accuracy by 43% on the NaturalQuestions-Open dataset with the MPT-7B instruction-tuned model over naive RAG.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 38 of its 50 references that are in the map. First posted on arXiv 10 Apr 2024.

Builds on

38 of 50 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May

Other references · 37

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  10. 2023A Survey of Large Language Modelscited by 2,537
  11. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  12. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
25 more
  1. 2020Reformer: The Efficient Transformercited by 1,658
  2. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  4. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  5. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  6. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  7. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  8. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  9. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  10. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  11. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  12. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  13. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  14. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  15. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  16. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  17. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  18. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 114
  19. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  20. 2023Attention Sorting Combats Recency Bias In Long Context Language Modelscited by 89
  21. 2022Parallel Context Windows for Large Language Modelscited by 81
  22. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  23. 2023Prompting Frameworks for Large Language Models: A Surveycited by 29
  24. 2023Scaling In-Context Demonstrations with Structured Attentioncited by 8
  25. 2023Position Interpolation Improves ALiBi Extrapolationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 6; 2026: 5.

15 papers.

Sort:

By area:

  1. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  2. 2024KBLaM: Knowledge Base augmented Language Modelcited by 16
  3. 2025CLaRa: Bridging Retrieval and Generation with Continuous Latent Reasoningcited by 9
  4. 2025AGRAG: Advanced Graph-based Retrieval-Augmented Generation for LLMscited by 7
  5. 2025Accelerating Adaptive Retrieval Augmented Generation via Instruction-Driven Representation Reduction of Retrieval Overlapscited by 6
  6. 2024G-RAG: Knowledge Expansion in Material Sciencecited by 2
  7. 2026MiniPIC: Flexible Position-Independent Caching in <100LOCcited by 2
  8. 2024RAG-Modulo: Solving Sequential Tasks using Experience, Critics, and Language Modelscited by 1
  9. 2025Knowledge-Grounded Agentic Large Language Models for Multi-Hazard Understanding from Reconnaissance Reportscited by 1
  10. 2025CacheFocus: Dynamic Cache Re-Positioning for Efficient Retrieval-Augmented Generationcited by 0
  11. 2025Accelerating Causal Network Discovery of Alzheimer Disease Biomarkers via Scientific Literature-based Retrieval Augmented Generationcited by 0
  12. 2026Towards Generalization of Block Attention via Automatic Segmentation and Block Distillationcited by 0
3 more
  1. 2026PARSER: Read in Parallel, Reason in Depth for Long-Context LLM Agentscited by 0
  2. 2026Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMscited by 0
  3. 2026Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Emulating Retrieval Augmented Generation via Prompt Engineering for Enhanced Long Context Comprehension in LLMs
  2. 2026Evaluating Prompt Engineering Techniques for RAG in Small Language Models: A Multi-Hop QA Approach
  3. 2025Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers
  4. 2024Prompt Perturbation in Retrieval-Augmented Generation based Large Language Models
  5. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG
  6. 2024In Defense of RAG in the Era of Long-Context Language Models
  7. 2024Inference Scaling for Long-Context Retrieval Augmented Generation
  8. 2024The Power of Noise: Redefining Retrieval for RAG Systems