mlsanityData to 7 Oct 2026

Data Curation Alone Can Stabilize In-context Learning

arXiv 2212.10378 ↗First posted 20 Dec 2022cs.CLcited by 58

Not on the map. Why?

Abstract (arXiv)

In-context learning (ICL) enables large language models (LLMs) to perform new tasks by prompting them with a sequence of training examples. However, it is known that ICL is very sensitive to the choice of training examples: randomly sampling examples from a training set leads to high variance in performance. In this paper, we show that carefully curating a subset of training data greatly stabilizes ICL performance without any other changes to the ICL algorithm (e.g., prompt retrieval or calibration). We introduce two methods to choose training subsets -- both score training examples individually, then select the highest-scoring ones. CondAcc scores a training example by its average dev-set ICL accuracy when combined with random training examples, while Datamodels learns linear regressors that estimate how the presence of each training example influences LLM outputs. Across five tasks and two LLMs, sampling from stable subsets selected by CondAcc and Datamodels improves average accuracy over sampling from the entire training set by 7.7% and 6.3%, respectively. Surprisingly, the stable subset examples are not especially diverse in content or low in perplexity, in contrast with other work suggesting that diversity and perplexity are important when prompting LLMs.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Builds on 38 of its 54 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

38 of 54 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  9. 2022Emergent Abilities of Large Language Modelscited by 2,483
  10. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  12. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
26 more
  1. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  2. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  3. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  4. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  5. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  6. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  7. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  8. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  9. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  10. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  11. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 617
  12. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  13. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  14. 2021True Few-Shot Learning with Language Modelscited by 343
  15. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  16. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  17. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 215
  18. 2022Active Example Selection for In-Context Learningcited by 195
  19. 2022Datamodels: Predicting Predictions from Training Datacited by 186
  20. 2019Structural Scaffolds for Citation Intent Classification in Scientific Publicationscited by 129
  21. 2020Cold-start Active Learning through Self-supervised Language Modelingcited by 128
  22. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  23. 2019Data Valuation using Reinforcement Learningcited by 115
  24. 2019Diverse mini-batch Active Learningcited by 88
  25. 2022On the Relation between Sensitivity and Accuracy in In-context Learningcited by 71
  26. 2022Complementary Explanations for Effective In-Context Learningcited by 71

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 35; 2025: 12; 2026: 2.

58 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2023Finding Support Examples for In-Context Learningcited by 104
  3. 2024ContextCite: Attributing Model Generation to Contextcited by 93
  4. 2024Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvementscited by 82
  5. 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplificationcited by 80
  6. 2023Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regressioncited by 47
  7. 2024IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representationscited by 37
  8. 2024Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplarscited by 37
  9. 2023How are Prompts Different in Terms of Sensitivity?cited by 34
  10. 2024Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMscited by 32
  11. 2024Smoothie: Label Free Language Model Routingcited by 31
  12. 2024A Survey of Table Reasoning with Large Language Modelscited by 29
38 more
  1. 2023IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Modelscited by 26
  2. 2023RetICL: Sequential Retrieval of In-Context Examples with Reinforcement Learningcited by 24
  3. 2024Beyond Performance: Quantifying and Mitigating Label Bias in LLMscited by 24
  4. 2024LLM Task Interference: An Initial Study on the Impact of Task-Switch in Conversational Historycited by 19
  5. 2024Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystackcited by 19
  6. 2024Effects of diversity incentives on sample diversity and downstream model performance in LLM-based text augmentationcited by 17
  7. 2024SynthDST: Synthetic Data is All You Need for Few-Shot Dialog State Trackingcited by 17
  8. 2024Comparing Specialised Small and General Large Language Models on Text Classification: 100 Labelled Samples to Achieve Break-Even Performancecited by 17
  9. 2024Reconstruct Your Previous Conversations! Comprehensively Investigating Privacy Leakage Risks in Conversations with GPT Modelscited by 16
  10. 2024Revisiting In-context Learning Inference Circuit in Large Language Modelscited by 16
  11. 2023A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomnesscited by 14
  12. 2023Improving Probability-based Prompt Selection Through Unified Evaluation and Analysiscited by 13
  13. 2024In-Context Example Ordering Guided by Label Distributionscited by 12
  14. 2024Addressing Order Sensitivity of In-Context Demonstration Examples in Causal Language Modelscited by 12
  15. 2024In-Context Learning Demonstration Selection via Influence Analysiscited by 11
  16. 2024Strategic Demonstration Selection for Improved Fairness in LLM In-Context Learningcited by 10
  17. 2024FastGAS: Fast Graph-based Annotation Selection for In-Context Learningcited by 8
  18. 2024LLMs vs Established Text Augmentation Techniques for Classification: When do the Benefits Outweight the Costs?cited by 8
  19. 2024Use Random Selection for Now: Investigation of Few-Shot Selection Strategies in LLM-based Text Augmentation for Classificationcited by 7
  20. 2024When Parts Are Greater Than Sums: Individual LLM Components Can Outperform Full Modelscited by 6
  21. 2025Enhancing Multimodal In-Context Learning for Image Classification through Coreset Optimizationcited by 6
  22. 2025The Role of Diversity in In-Context Learning for Large Language Modelscited by 6
  23. 2024DETAIL: Task DEmonsTration Attribution for Interpretable In-context Learningcited by 5
  24. 2024EXPLORA: Efficient Exemplar Subset Selection for Complex Reasoningcited by 5
  25. 2025CrowdLLM: Building LLM-Based Digital Populations Augmented with Generative Modelscited by 5
  26. 2023The Role of Data Curation in Image Captioningcited by 4
  27. 2024Automatic Combination of Sample Selection Strategies for Few-Shot Learningcited by 4
  28. 2024LLMs for Targeted Sentiment in News Headlines: Exploring the Descriptive-Prescriptive Dilemmacited by 3
  29. 2024DemoShapley: Valuation of Demonstrations for In-Context Learningcited by 3
  30. 2025Analyzing Similarity Metrics for Data Selection for Language Model Pretrainingcited by 3
  31. 2024Pelican Soup Framework: A Theoretical Framework for Language Model Capabilitiescited by 2
  32. 2024A critical review of methods and challenges in large language modelscited by 2
  33. 2025Monte Carlo Sampling for Analyzing In-Context Examplescited by 2
  34. 2025Stability of In-Context Learning: A Spectral Coverage Perspectivecited by 2
  35. 2024On Sensitivity of Learning with Limited Labelled Data to the Effects of Randomness: Impact of Interactions and Systematic Choicescited by 1
  36. 2024Augmenting In-Context-Learning in LLMs via Automatic Data Labeling and Refinementcited by 1
  37. 2024Effective Self-Mining of In-Context Examples for Unsupervised Machine Translation with LLMscited by 1
  38. 2025Retrieval augmented generation based dynamic prompting for few-shot biomedical named entity recognition using large language modelscited by 1

50 most cited of 58 All 58 (JSON)