mlsanityData to 14 Oct 2026

Learning To Retrieve Prompts for In-Context Learning

arXiv 2112.08633 ↗First posted 16 Dec 2021cs.CL cs.LGcited by 648

Not on the map. Why?

Cited by 74 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In-context learning is a recent paradigm in natural language understanding, where a large pre-trained language model (LM) observes a test instance and a few training examples as its input, and directly decodes the output without any update to its parameters. However, performance has been shown to strongly depend on the selected training examples (termed prompt). In this work, we propose an efficient method for retrieving prompts for in-context learning using annotated data and a LM. Given an input-output pair, we estimate the probability of the output given the input and a candidate training example as the prompt, and label training examples as positive or negative based on this probability. We then train an efficient dense retriever from this data, which is used to retrieve training examples as prompts at test time. We evaluate our approach on three sequence-to-sequence tasks where language utterances are mapped to meaning representations, and find that it substantially outperforms prior work and multiple baselines across the board.

Key facts. Cited by 648 papers in the map (data through 14 Oct 2026). Builds on 43 of its 63 references that are in the map. First posted on arXiv 16 Dec 2021.

Builds on

43 of 63 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  9. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  10. 2017Billion-scale similarity search with GPUscited by 2,497
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  12. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
31 more
  1. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  2. 2019Language Models as Knowledge Bases?cited by 1,936
  3. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  4. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  5. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  6. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  7. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  8. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  9. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  10. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  11. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  12. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  13. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 695
  14. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 413
  15. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 377
  16. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  17. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 263
  18. 2017Efficient Natural Language Response Suggestion for Smart Replycited by 229
  19. 2020Nearest Neighbor Machine Translationcited by 209
  20. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 165
  21. 2021Constrained Language Models Yield Few-Shot Semantic Parserscited by 163
  22. 2020MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmarkcited by 158
  23. 2020Break It Down: A Question Understanding Benchmarkcited by 131
  24. 2020Task-Oriented Dialogue as Dataflow Synthesiscited by 122
  25. 2021Case-based Reasoning for Natural Language Queries over Knowledge Basescited by 107
  26. 2021BERTese: Learning to Speak to BERTcited by 82
  27. 2020A Mathematical Exploration of Why Language Models Help Solve Downstream Taskscited by 72
  28. 2021Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attentioncited by 40
  29. 2021Controllable Semantic Parsing via Retrieval Augmentationcited by 35
  30. 2021The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Designcited by 33
  31. 2021Question Decomposition with Dependency Graphscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 59; 2023: 190; 2024: 199; 2025: 109; 2026: 91.

648 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,576
  2. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  3. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  4. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 876
  5. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  6. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  7. 2022A Survey on In-context Learningcited by 721
  8. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  9. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
  10. 2023A Comprehensive Overview of Large Language Modelscited by 648
  11. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  12. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 452
38 more
  1. 2023Cognitive Architectures for Language Agentscited by 429
  2. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 388
  3. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Modelscited by 379
  4. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 333
  5. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  6. 2023Can We Edit Factual Knowledge by In-Context Learning?cited by 285
  7. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 267
  8. 2022Reasoning with Language Model Prompting: A Surveycited by 252
  9. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 245
  10. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 243
  11. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 233
  12. 2024Generative Representational Instruction Tuningcited by 233
  13. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 232
  14. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 228
  15. 2022Active Example Selection for In-Context Learningcited by 196
  16. 2023Recommender Systems in the Era of Large Language Models (LLMs)cited by 189
  17. 2022Can Foundation Models Wrangle Your Data?cited by 175
  18. 2022In-context Examples Selection for Machine Translationcited by 172
  19. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 161
  20. 2022Language Models of Code are Few-Shot Commonsense Learnerscited by 160
  21. 2023GPT-NER: Named Entity Recognition via Large Language Modelscited by 160
  22. 2022Instruction Induction: From Few Examples to Natural Language Task Descriptionscited by 152
  23. 2023What Makes Good Examples for Visual In-Context Learning?cited by 150
  24. 2023Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learningcited by 149
  25. 2023Progressive-Hint Prompting Improves Reasoning in Large Language Modelscited by 147
  26. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 146
  27. 2023Compositional Exemplars for In-context Learningcited by 144
  28. 2023AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Webcited by 144
  29. 2023Unified Demonstration Retriever for In-Context Learningcited by 142
  30. 2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 139
  31. 2022Internet-augmented language models through few-shot prompting for open-domain question answeringcited by 131
  32. 2023EvoPrompting: Language Models for Code-Level Neural Architecture Searchcited by 131
  33. 2023Recommender AI Agent: Integrating Large Language Models for Interactive Recommendationscited by 131
  34. 2024VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Modelscited by 131
  35. 2022Text and Patterns: For Effective Chain of Thought, It Takes Two to Tangocited by 128
  36. 2023DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Modelscited by 128
  37. 2024DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoningcited by 125
  38. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124

50 most cited of 648 All 648 (JSON)