mlsanityData to 7 Oct 2026

Understanding In-Context Learning via Supportive Pretraining Data

arXiv 2306.15091 ↗First posted 26 Jun 2023cs.CLcited by 41

Not on the map. Why?

Abstract (arXiv)

In-context learning (ICL) improves language models' performance on a variety of NLP tasks by simply demonstrating a handful of examples at inference time. It is not well understood why ICL ability emerges, as the model has never been specifically trained on such demonstrations. Unlike prior work that explores implicit mechanisms behind ICL, we study ICL via investigating the pretraining data. Specifically, we first adapt an iterative, gradient-based approach to find a small subset of pretraining data that supports ICL. We observe that a continued pretraining on this small subset significantly improves the model's ICL ability, by up to 18%. We then compare the supportive subset constrastively with random subsets of pretraining data and discover: (1) The supportive pretraining data to ICL do not have a higher domain relevance to downstream tasks. (2) The supportive pretraining data have a higher mass of rarely occurring, long-tail tokens. (3) The supportive pretraining data are challenging examples where the information gain from long-range context is below average, indicating learning to incorporate difficult long-range context encourages ICL. Our work takes a first step towards understanding ICL via analyzing instance-level pretraining data. Our insights have a potential to enhance the ICL ability of language models by actively guiding the construction of pretraining data in the future.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 33 of its 44 references that are in the map. First posted on arXiv 26 Jun 2023.

Builds on

33 of 44 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  10. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  11. 2019Language Models as Knowledge Bases?cited by 1,922
  12. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
21 more
  1. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  2. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  3. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  4. 2022In-context Learning and Induction Headscited by 930
  5. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  6. 2022Transformers learn in-context by gradient descentcited by 714
  7. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  8. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  9. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  10. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  11. 2023Larger language models do in-context learning differentlycited by 386
  12. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  13. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 149
  14. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  15. 2022On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Modelcited by 78
  16. 2019TWEETQA: A Social Media Focused Question Answering Datasetcited by 55
  17. 2023Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runscited by 52
  18. 2022On the Blind Spots of Model-Based Evaluation Metrics for Text Generationcited by 48
  19. 2023Pre-Training to Learn in Contextcited by 41
  20. 2022ORCA: Interpreting Prompted Language Models via Locating Supporting Data Evidence in the Ocean of Pretraining Datacited by 34
  21. 2022Robustness of Demonstration-based Learning Under Limited Data Scenariocited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 19; 2025: 11; 2026: 7.

41 papers.

Sort:

By area:

  1. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  2. 2024Can large language models explore in-context?cited by 75
  3. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  4. 2024From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examplescited by 55
  5. 2023MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learningcited by 46
  6. 2024Scalable Influence and Fact Tracing for Large Language Model Pretrainingcited by 40
  7. 2024Unveiling the Generalization Power of Fine-Tuned Large Language Modelscited by 35
  8. 2024SelectIT: Selective Instruction Tuning for LLMs via Uncertainty-Aware Self-Reflectioncited by 34
  9. 2024The broader spectrum of in-context learningcited by 33
  10. 2024Do Language Models Exhibit Human-like Structural Priming Effects?cited by 21
  11. 2023Supervised Knowledge Makes Large Language Models Better In-context Learnerscited by 20
  12. 2024BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignmentcited by 20
29 more
  1. 2023Do pretrained Transformers Learn In-Context by Gradient Descent?cited by 18
  2. 2024Unlearning Traces the Influential Training Data of Language Modelscited by 18
  3. 2024Revisiting In-context Learning Inference Circuit in Large Language Modelscited by 16
  4. 2024Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet?cited by 11
  5. 2024Token-based Decision Criteria Are Suboptimal in In-context Learningcited by 9
  6. 2026Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuningcited by 9
  7. 2024Influential Language Data Selection via Gradient Trajectory Pursuitcited by 7
  8. 2025ZEUS: Zero-shot Embeddings for Unsupervised Separation of Tabular Datacited by 6
  9. 2025Scaling sparse feature circuit finding for in-context learningcited by 5
  10. 2024NoisyICL: A Little Noise in Model Parameters Calibrates In-context Learningcited by 4
  11. 2024One size doesn't fit all: Predicting the Number of Examples for In-Context Learningcited by 4
  12. 2025Unlocking In-Context Learning for Natural Datasets Beyond Language Modellingcited by 4
  13. 2026Detecting and Suppressing Reward Hacking with Gradient Fingerprintscited by 4
  14. 2026OpenRFM: Dissecting Relational In-Context Learningcited by 4
  15. 2024From Unstructured Data to In-Context Learning: Exploring What Tasks Can Be Learned and Whencited by 3
  16. 2024No Size Fits All: The Perils and Pitfalls of Leveraging LLMs Vary with Company Sizecited by 3
  17. 2024Identifying and Analyzing Performance-Critical Tokens in Large Language Modelscited by 2
  18. 2025StaICC: Standardized Evaluation for Classification Task in In-context Learningcited by 2
  19. 2025Tree of Preferences for Diversified Recommendationcited by 2
  20. 2025Metareasoning constraints couple narratives, affect and cognitioncited by 1
  21. 2025ProDS: Preference-oriented Data Selection for Instruction Tuningcited by 1
  22. 2025One Period to Rule Them All: Identifying Critical Learning Periods in Deep Networkscited by 1
  23. 2025Learning-to-Context Slope: Evaluating In-Context Learning Effectiveness Beyond Performance Illusionscited by 1
  24. 2025InSQuAD: In-Context Learning for Efficient Retrieval via Submodular Mutual Information to Enforce Quality and Diversitycited by 1
  25. 2026Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottleneckscited by 1
  26. 2025Rewriting History: A Recipe for Interventional Analyses to Study Data Effects on Model Behaviorcited by 0
  27. 2026TRUE: A Trustworthy Unified Explanation Framework for Large Language Model Reasoningcited by 0
  28. 2026Combining Retrieval-Augmented Text Generation with LLMs for Reading Content Recommendationscited by 0
  29. 2026Shortcut Before Circuit: Document Statistics Time In-Context Conflict Resolutioncited by 0