mlsanityData to 14 Oct 2026

Learning to Retrieve In-Context Examples for Large Language Models

arXiv 2307.07164 ↗First posted 14 Jul 2023cs.CL cs.IRcited by 60

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have demonstrated their ability to learn in-context, allowing them to perform various tasks based on a few input-output examples. However, the effectiveness of in-context learning is heavily reliant on the quality of the selected examples. In this paper, we propose a novel framework to iteratively train dense retrievers that can identify high-quality in-context examples for LLMs. Our framework initially trains a reward model based on LLM feedback to evaluate the quality of candidate examples, followed by knowledge distillation to train a bi-encoder based dense retriever. Our experiments on a suite of 30 tasks demonstrate that our framework significantly enhances in-context learning performance. Furthermore, we show the generalization ability of our framework to unseen tasks during training. An in-depth analysis reveals that our model improves performance by retrieving examples with similar patterns, and the gains are consistent across LLMs of varying sizes. The code and data are available at https://github.com/microsoft/LMOps/tree/main/llmretriever .

Key facts. Cited by 60 papers in the map (data through 14 Oct 2026). Builds on 41 of its 60 references that are in the map. First posted on arXiv 14 Jul 2023.

Builds on

41 of 60 references.

References · 41

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
29 more
  1. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  2. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  4. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  5. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  7. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  8. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  9. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  10. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  11. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  12. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  13. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 849
  14. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  15. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  16. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  17. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  18. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 402
  19. 2022Active Example Selection for In-Context Learningcited by 196
  20. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 176
  21. 2020DART: Open-Domain Structured Data Record to Text Generationcited by 145
  22. 2023Compositional Exemplars for In-context Learningcited by 144
  23. 2023Unified Demonstration Retriever for In-Context Learningcited by 142
  24. 2022Structured Prompting: Scaling In-Context Learning to 1,000 Examplescited by 81
  25. 2021Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challengecited by 80
  26. 2023UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluationcited by 75
  27. 2023Dr.ICL: Demonstration-Retrieved In-context Learningcited by 72
  28. 2019Semantic Noise Matters for Neural Natural Language Generationcited by 64
  29. 2019This Email Could Save Your Life: Introducing the Task of Email Subject Line Generationcited by 58

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 20; 2025: 17; 2026: 14.

60 papers.

Sort:

By area:

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  2. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 876
  3. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 354
  4. 2023Efficient Large Language Models: A Surveycited by 186
  5. 2024DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoningcited by 125
  6. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 70
  7. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  8. 2024Training-Free Long-Context Scaling of Large Language Modelscited by 62
  9. 2023PromptCBLUE: A Chinese Prompt Tuning Benchmark for the Medical Domaincited by 40
  10. 2023Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?cited by 34
  11. 2024INMS: Memory Sharing for Large Language Model based Agentscited by 31
  12. 2024C-RAG: Certified Generation Risks for Retrieval-Augmented Language Modelscited by 27
38 more
  1. 2023Fewer is More: Boosting LLM Reasoning with Reinforced Context Pruningcited by 26
  2. 2024Sequential Large Language Model-Based Hyper-parameter Optimizationcited by 24
  3. 2025True Multimodal In-Context Learning Needs Attention to the Visual Contextcited by 16
  4. 2024Logical Consistency of Large Language Models in Fact-checkingcited by 15
  5. 2024VURF: A General-purpose Reasoning and Self-refinement Framework for Video Understandingcited by 13
  6. 2025MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learningcited by 13
  7. 2025Time Series Language Model for Descriptive Caption Generationcited by 11
  8. 2024LLaMandement: Large Language Models for Summarization of French Legislative Proposalscited by 9
  9. 2025Enhancing Training Data Attribution with Representational Optimizationcited by 8
  10. 2023Lever LM: Configuring In-Context Sequence to Lever Large Vision Language Modelscited by 7
  11. 2025Advancing Multimodal In-Context Learning in Large Vision-Language Models with Task-aware Demonstrationscited by 7
  12. 2025Optimizing Case-Based Reasoning System for Functional Test Script Generation with Large Language Modelscited by 7
  13. 2026MemCollab: Cross-Model Memory Collaboration via Contrastive Trajectory Distillationcited by 7
  14. 2023GistScore: Learning Better Representations for In-Context Example Selection with Gist Bottleneckscited by 6
  15. 2025DICE: Dynamic In-Context Example Selection in LLM Agents via Efficient Knowledge Transfercited by 5
  16. 2024Large Scale Generative AI Text Applied to Sports and Musiccited by 4
  17. 2024Evaluating and Safeguarding the Adversarial Robustness of Retrieval-Based In-Context Learningcited by 4
  18. 2024RAGSys: Item-Cold-Start Recommender as RAG Systemcited by 4
  19. 2025Being Strong Progressively! Enhancing Knowledge Distillation of Large Language Models through a Curriculum Learning Frameworkcited by 4
  20. 2024Vector Retrieval with Similarity and Diversity: How Hard Is It?cited by 3
  21. 2023The Tyranny of Possibilities in the Design of Task-Oriented LLM Systems: A Scoping Surveycited by 2
  22. 2024From Cool Demos to Production-Ready FMware: Core Challenges and a Technology Roadmapcited by 2
  23. 2025RIDE: Enhancing Large Language Model Alignment through Restyled In-Context Learning Demonstration Exemplarscited by 2
  24. 2025InPhyRe Discovers: Large Multimodal Models Struggle in Inductive Physical Reasoningcited by 2
  25. 2025Fine-Tuned In-Context Learners for Efficient Adaptationcited by 2
  26. 2024Physics Event Classification Using Large Language Modelscited by 1
  27. 2024ICLERB: In-Context Learning Embedding and Reranker Benchmarkcited by 1
  28. 2025Examples as the Prompt: A Scalable Approach for Efficient LLM Adaptation in E-Commercecited by 1
  29. 2025Learning-to-Context Slope: Evaluating In-Context Learning Effectiveness Beyond Performance Illusionscited by 1
  30. 2026Document-Authored Control-Signal Impersonation: A Low-Cost Indirect Prompt Attack on RAG Safety Boundariescited by 1
  31. 2026GRIP: Feedback-Guided Prompt Retrieval for Large Multimodal Modelscited by 1
  32. 2026ContextFlow: In-Context Flow Matching for Robot Manipulationcited by 1
  33. 2024Guiding ChatGPT to Generate Salient Domain Summariescited by 0
  34. 2024Distributed In-Context Learning under Non-IID Among Clientscited by 0
  35. 2025Medical Interpretability and Knowledge Maps of Large Language Modelscited by 0
  36. 2025Building Specialized Software-Assistant ChatBot with Graph-Based Retrieval-Augmented Generationcited by 0
  37. 2025Context-Picker: Dynamic context selection using multi-stage reinforcement learningcited by 0
  38. 2025From Retrieval to Reasoning: A Framework for Cyber Threat Intelligence NER with Explicit and Adaptive Instructionscited by 0

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Large Language Models Know What Makes Exemplary Contexts
  2. 2024Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning
  3. 2024Efficient LLM Context Distillation
  4. 2024Can Large Language Models Understand Context?
  5. 2023In-Context Explainers: Harnessing LLMs for Explaining Black Box Models
  6. 2024In-Context Learning with Reinforcement Learning for Incomplete Utterance Rewriting
  7. 2024In-Context Language Learning: Architectures and Algorithms
  8. 2024RARe: Retrieval Augmented Retrieval with In-Context Examples