mlsanityData to 14 Oct 2026

In-context Learning with Retrieved Demonstrations for Language Models: A Survey

arXiv 2401.11624 ↗First posted 21 Jan 2024cs.CL cs.AI cs.IRcited by 70

Not on the map. Why?

Abstract (arXiv)

Language models, especially pre-trained large language models, have showcased remarkable abilities as few-shot in-context learners (ICL), adept at adapting to new tasks with just a few demonstrations in the input context. However, the model's ability to perform ICL is sensitive to the choice of the few-shot demonstrations. Instead of using a fixed set of demonstrations, one recent development is to retrieve demonstrations tailored to each input query. The implementation of demonstration retrieval is relatively straightforward, leveraging existing databases and retrieval systems. This not only improves the efficiency and scalability of the learning process but also has been shown to reduce biases inherent in manual example selection. In light of the encouraging results and growing research in ICL with retrieved demonstrations, we conduct an extensive review of studies in this area. In this survey, we discuss and compare different design choices for retrieval models, retrieval training procedures, and inference algorithms.

Key facts. Cited by 70 papers in the map (data through 14 Oct 2026). Builds on 132 of its 175 references that are in the map. First posted on arXiv 21 Jan 2024.

Builds on

132 of 175 references.

References · 132

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  10. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  11. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  12. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
38 more
  1. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  12. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  13. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  14. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  15. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  16. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  17. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  18. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  19. 2023A Survey of Large Language Modelscited by 2,576
  20. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  21. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  22. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  23. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  24. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  25. 2019Language Models as Knowledge Bases?cited by 1,936
  26. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  27. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  28. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  29. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  30. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  31. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  32. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  33. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  34. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  35. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  36. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  37. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  38. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135

50 most cited of 132 All 132 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 27; 2025: 35; 2026: 8.

70 papers.

Sort:

By area:

  1. 2024LLaMEA: A Large Language Model Evolutionary Algorithm for Automatically Generating Metaheuristicscited by 97
  2. 2024What Makes Multimodal In-Context Learning Work?cited by 56
  3. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 54
  4. 2024Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTScited by 54
  5. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 51
  6. 2025UFO2: The Desktop AgentOScited by 41
  7. 2024Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplarscited by 37
  8. 2024Mixture of In-Context Prompters for Tabular PFNscited by 33
  9. 2024AMOR: A Recipe for Building Adaptable Modular Knowledge Agents Through Process Feedbackcited by 25
  10. 2025TACO: Enhancing Multimodal In-context Learning via Task Mapping-Guided Sequence Configurationcited by 23
  11. 2025From Few to Many: Self-Improving Many-Shot Reasoners Through Iterative Optimization and Generationcited by 21
  12. 2024CEB: Compositional Evaluation Benchmark for Fairness in Large Language Modelscited by 18
38 more
  1. 2024Retrieval Meets Reasoning: Even High-school Textbook Knowledge Benefits Multimodal Reasoningcited by 14
  2. 2024CTINexus: Automatic Cyber Threat Intelligence Knowledge Graph Construction Using Large Language Modelscited by 14
  3. 2024Assessing Empathy in Large Language Models with Real-World Physician-Patient Interactionscited by 13
  4. 2025IC-Cache: Efficient Large Language Model Serving via In-context Cachingcited by 13
  5. 2025Problem-Solving Logic Guided Curriculum In-Context Learning for LLMs Complex Reasoningcited by 12
  6. 2025Mimic In-Context Learning for Multimodal Taskscited by 12
  7. 2024Improving Demonstration Diversity by Human-Free Fusing for Text-to-SQLcited by 10
  8. 2024Hyperband-based Bayesian Optimization for Black-box Prompt Selectioncited by 10
  9. 2025Federated Retrieval Augmented Generation for Multi-Product Question Answeringcited by 10
  10. 2024SPL: A Socratic Playground for Learning Powered by Large Language Modelcited by 9
  11. 2024Divide, Reweight, and Conquer: A Logit Arithmetic Approach for In-Context Learningcited by 9
  12. 2026Demo-ICL: In-Context Learning for Procedural Video Knowledge Acquisitioncited by 8
  13. 2024XAMPLER: Learning to Retrieve Cross-Lingual In-Context Examplescited by 5
  14. 2024Mixtures of In-Context Learnerscited by 5
  15. 2024MICAS: Multi-grained In-Context Adaptive Sampling for 3D Point Cloud Processingcited by 5
  16. 2025Learning to Select In-Context Demonstration Preferred by Large Language Modelcited by 5
  17. 2024One size doesn't fit all: Predicting the Number of Examples for In-Context Learningcited by 4
  18. 2025Why is prompting hard? Understanding prompts on binary sequence predictorscited by 4
  19. 2025PICASO: Permutation-Invariant Context Composition with State Space Modelscited by 4
  20. 2025DCG-SQL: Enhancing In-Context Learning for Text-to-SQL with Deep Contextual Schema Link Graphcited by 4
  21. 2025Retrieval-Augmented Generation as Noisy In-Context Learning: A Unified Theory and Risk Boundscited by 4
  22. 2025Neural Network Reprogrammability: A Unified Theme on Model Reprogramming, Prompt Tuning, and Prompt Instructioncited by 4
  23. 2024In-Context Transfer Learning: Demonstration Synthesis by Transferring Similar Taskscited by 3
  24. 2024DemoShapley: Valuation of Demonstrations for In-Context Learningcited by 3
  25. 2024Comparative Analysis of Demonstration Selection Algorithms for LLM In-Context Learningcited by 3
  26. 2025Post-Incorporating Code Structural Knowledge into Pretrained Models via ICL for Code Translationcited by 3
  27. 2025RICo: Refined In-Context Contribution for Automatic Instruction-Tuning Data Selectioncited by 3
  28. 2025Retrieval-Augmented Review Generation for Poisoning Recommender Systemscited by 3
  29. 2026Modality-free Graph In-context Alignmentcited by 3
  30. 2025MateICL: Mitigating Attention Dispersion in Large-Scale In-Context Learningcited by 2
  31. 2025Spoken Language Understanding on Unseen Tasks With In-Context Learningcited by 2
  32. 2025Delta-KNN: Improving Demonstration Selection in In-Context Learning for Alzheimer's Disease Detectioncited by 2
  33. 2025Distilling Many-Shot In-Context Learning into a Cheat Sheetcited by 2
  34. 2024E-ICL: Enhancing Fine-Grained Emotion Recognition through the Lens of Prototype Theorycited by 1
  35. 2024DoubleDipper: Improving Long-Context LLMs via Context Recyclingcited by 1
  36. 2025Explanation based In-Context Demonstrations Retrieval for Multilingual Grammatical Error Correctioncited by 1
  37. 2025LLMIA: An Out-of-the-Box Index Advisor via In-Context Learning with LLMscited by 1
  38. 2025Making Large Language Models Better Reasoners with Orchestrated Streaming Experiencescited by 1

50 most cited of 70 All 70 (JSON)