mlsanityData to 7 Oct 2026

What Makes Good In-Context Examples for GPT-3?

arXiv 2101.06804 ↗First posted 17 Jan 2021cs.CLcited by 1,184

Not on the map. Why?

Cited by 115 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

GPT-3 has attracted lots of attention due to its superior performance across a wide range of NLP tasks, especially with its powerful and versatile in-context few-shot learning ability. Despite its success, we found that the empirical results of GPT-3 depend heavily on the choice of in-context examples. In this work, we investigate whether there are more effective strategies for judiciously selecting in-context examples (relative to random sampling) that better leverage GPT-3's few-shot capabilities. Inspired by the recent success of leveraging a retrieval module to augment large-scale neural network models, we propose to retrieve examples that are semantically-similar to a test sample to formulate its corresponding prompt. Intuitively, the in-context examples selected with such a strategy may serve as more informative inputs to unleash GPT-3's extensive knowledge. We evaluate the proposed approach on several natural language understanding and generation benchmarks, where the retrieval-based prompt selection approach consistently outperforms the random baseline. Moreover, it is observed that the sentence encoders fine-tuned on task-related datasets yield even more helpful retrieval results. Notably, significant gains are observed on tasks such as table-to-text generation (41.9% on the ToTTo dataset) and open-domain question answering (45.5% on the NQ dataset). We hope our investigation could help understand the behaviors of GPT-3 and large-scale pre-trained LMs in general and enhance their few-shot capabilities.

Key facts. Cited by 1,184 papers in the map (data through 7 Oct 2026). Builds on 38 of its 54 references that are in the map. First posted on arXiv 17 Jan 2021.

Builds on

38 of 54 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
26 more
  1. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  2. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  3. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  4. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  6. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  7. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  8. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  9. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  10. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  11. 2020ToTTo: A Controlled Table-To-Text Generation Datasetcited by 267
  12. 2020Nearest Neighbor Machine Translationcited by 209
  13. 2018Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfercited by 191
  14. 2020Generation-Augmented Retrieval for Open-domain Question Answeringcited by 186
  15. 2017Faithful to the Original: Fact Aware Neural Abstractive Summarizationcited by 141
  16. 2019Handling Divergent Reference Texts when Evaluating Table-to-Text Generationcited by 119
  17. 2017Generating Sentences by Editing Prototypescited by 88
  18. 2018Retrieve and Refine: Improved Sequence Generation Models For Dialoguecited by 82
  19. 2018A Retrieve-and-Edit Framework for Predicting Structured Outputscited by 73
  20. 2018Skeleton-to-Response: Dialogue Generation Guided by Retrieval Memorycited by 49
  21. 2018Response Generation by Context-aware Prototype Editingcited by 40
  22. 2020BERT-kNN: Adding a kNN Search Component to Pretrained Language Models for Better QAcited by 36
  23. 2019Text Generation with Exemplar-based Adaptive Decodingcited by 28
  24. 2020Explaining and Improving Model Behavior with k Nearest Neighbor Representationscited by 27
  25. 2020Example-Based Named Entity Recognitioncited by 19
  26. 2016Two are Better than One: An Ensemble of Retrieval- and Generation-Based Dialog Systemscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 30; 2022: 118; 2023: 380; 2024: 342; 2025: 193; 2026: 121.

1,184 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  3. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  8. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  9. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  10. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  11. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  12. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
38 more
  1. 2022A Survey on In-context Learningcited by 717
  2. 2021GPT Understands, Toocited by 715
  3. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  4. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  5. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  6. 2023A Comprehensive Overview of Large Language Modelscited by 639
  7. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  8. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  9. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  10. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  11. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  12. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  13. 2023Cognitive Architectures for Language Agentscited by 422
  14. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  15. 2022Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)cited by 396
  16. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  17. 2022Using cognitive psychology to understand GPT-3cited by 389
  18. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  19. 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 378
  20. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  21. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
  22. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  23. 2021True Few-Shot Learning with Language Modelscited by 343
  24. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  25. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  26. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 319
  27. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 316
  28. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  29. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  30. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  31. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  32. 2023Can We Edit Factual Knowledge by In-Context Learning?cited by 284
  33. 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 265
  34. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  35. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 247
  36. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  37. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 240
  38. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240

50 most cited of 1,184 All 1,184 (JSON)