mlsanityData to 7 Oct 2026

Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP

arXiv 2212.14024 ↗First posted 28 Dec 2022cs.CL cs.IRcited by 316

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Retrieval-augmented in-context learning has emerged as a powerful approach for addressing knowledge-intensive tasks using frozen language models (LM) and retrieval models (RM). Existing work has combined these in simple "retrieve-then-read" pipelines in which the RM retrieves passages that are inserted into the LM prompt. To begin to fully realize the potential of frozen LMs and RMs, we propose Demonstrate-Search-Predict (DSP), a framework that relies on passing natural language texts in sophisticated pipelines between an LM and an RM. DSP can express high-level programs that bootstrap pipeline-aware demonstrations, search for relevant passages, and generate grounded predictions, systematically breaking down problems into small transformations that the LM and RM can handle more reliably. We have written novel DSP programs for answering questions in open-domain, multi-hop, and conversational settings, establishing in early evaluations new state-of-the-art in-context learning results and delivering 37-120%, 8-39%, and 80-290% relative gains against the vanilla LM (GPT-3.5), a standard retrieve-then-read pipeline, and a contemporaneous self-ask pipeline, respectively. We release DSP at https://github.com/stanfordnlp/dsp

Key facts. Cited by 316 papers in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 51 of its 63 references that are in the map. First posted on arXiv 28 Dec 2022.

Builds on

51 of 63 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Models5 Aug
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May

Other references · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  10. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  11. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  12. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
37 more
  1. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  2. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  3. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  4. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  5. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  6. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  7. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  8. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  9. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  10. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  11. 2022Large Language Models Can Self-Improvecited by 656
  12. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  13. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  14. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  15. 2021True Few-Shot Learning with Language Modelscited by 343
  16. 2022Prompting GPT-3 To Be Reliablecited by 258
  17. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 235
  18. 2020HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verificationcited by 199
  19. 2020Answering Complex Open-Domain Questions with Multi-Hop Dense Retrievalcited by 160
  20. 2020Open-Domain Question Answering Goes Conversational via Question Rewritingcited by 147
  21. 2022Internet-augmented language models through few-shot prompting for open-domain question answeringcited by 131
  22. 2019Multi-hop Reading Comprehension through Question Decomposition and Rescoringcited by 119
  23. 2022Rationale-Augmented Ensembles in Language Modelscited by 111
  24. 2022Language Model Cascadescited by 104
  25. 2022PLAID: An Efficient Engine for Late Interaction Retrievalcited by 95
  26. 2022On Decoding Strategies for Neural Text Generatorscited by 89
  27. 2022RankGen: Improving Text Generation with Large Ranking Modelscited by 72
  28. 2020Relevance-guided Supervision for OpenQA with ColBERTcited by 71
  29. 2022Recitation-Augmented Language Modelscited by 65
  30. 2021Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrievalcited by 63
  31. 2022FiD-Light: Efficient and Effective Retrieval-Augmented Text Generationcited by 63
  32. 2021Hindsight: Posterior-guided training of retrievers for improved open-ended generationcited by 38
  33. 2022Standing on the Shoulders of Giant Frozen Language Modelscited by 33
  34. 2022RoMQA: A Benchmark for Robust, Multi-evidence, Multi-answer Question Answeringcited by 19
  35. 2022Question rewriting? Assessing its importance for conversational question answeringcited by 15
  36. 2022Few-Shot Anaphora Resolution in Scientific Protocols via Mixtures of In-Context Expertscited by 10
  37. 2022SCAI-QReCC Shared Task on Conversational Question Answeringcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 80; 2024: 105; 2025: 88; 2026: 43.

316 papers.

Sort:

By area:

  1. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  2. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  3. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,411
  4. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  5. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  6. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  7. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  8. 2023Active Retrieval Augmented Generationcited by 506
  9. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  10. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 437
  11. 2023Cognitive Architectures for Language Agentscited by 422
  12. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 408
38 more
  1. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  2. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  3. 2023Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergycited by 343
  4. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  5. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 308
  6. 2023MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questionscited by 292
  7. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
  8. 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 270
  9. 2023Challenges and Applications of Large Language Modelscited by 251
  10. 2024TextGrad: Automatic "Differentiation" via Textcited by 250
  11. 2023Query Rewriting for Retrieval-Augmented Large Language Modelscited by 211
  12. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 200
  13. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 176
  14. 2024Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Modelscited by 150
  15. 2023PaperQA: Retrieval-Augmented Generative Agent for Scientific Researchcited by 143
  16. 2023ChatDB: Augmenting LLMs with Databases as Their Symbolic Memorycited by 136
  17. 2023Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generationcited by 128
  18. 2024xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Tokencited by 123
  19. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  20. 2023Large Language Models as Analogical Reasonerscited by 107
  21. 2024LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMscited by 104
  22. 2023Lift Yourself Up: Retrieval-augmented Text Generation with Self Memorycited by 103
  23. 2023Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakescited by 101
  24. 2024InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationalescited by 100
  25. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 97
  26. 2023Guiding Large Language Models via Directional Stimulus Promptingcited by 94
  27. 2023Pearl: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrieverscited by 91
  28. 2023Answering Questions by Meta-Reasoning over Multiple Chains of Thoughtcited by 87
  29. 2023Diagnostic Reasoning Prompts Reveal the Potential for Large Language Model Interpretability in Medicinecited by 83
  30. 2023ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agentcited by 78
  31. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Modelscited by 77
  32. 2023Context-faithful Prompting for Large Language Modelscited by 76
  33. 2024AutoFlow: Automated Workflow Generation for Large Language Model Agentscited by 76
  34. 2023Search-in-the-Chain: Interactively Enhancing Large Language Models with Search for Knowledge-intensive Taskscited by 73
  35. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
  36. 2023Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-Incited by 71
  37. 2023Self-Knowledge Guided Retrieval Augmentation for Large Language Modelscited by 69
  38. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 69

50 most cited of 316 All 316 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond the Parameters: A Technical Survey of Contextual Enrichment in Large Language Models: From In-Context Prompting to Causal Retrieval-Augmented Generation
  2. 2024Demonstration Notebook: Finding the Most Suited In-Context Learning Example from Interactions
  3. 2026To Memorize or to Retrieve: Scaling the Interaction Between Pretraining and Retrieval
  4. 2023Dr.ICL: Demonstration-Retrieved In-context Learning
  5. 2025GRAD: Generative Retrieval-Aligned Demonstration Sampler for Efficient Few-Shot Reasoning
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Training
  7. 2023Meta-training with Demonstration Retrieval for Efficient Few-shot Learning
  8. 2024Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning