mlsanityData to 14 Oct 2026

A Framework for Quantifying How Pre-Training and Context Benefit In-Context Learning

arXiv 2510.22594 ↗First posted 26 Oct 2025cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Pre-trained large language models have demonstrated a strong ability to learn from context, known as in-context learning (ICL). Despite a surge of recent applications that leverage such capabilities, it is by no means clear, at least theoretically, how the ICL capabilities arise, and in particular, what is the precise role played by key factors such as pre-training procedure as well as context construction. In this work, we propose a new framework to analyze the ICL performance, for a class of realistic settings, which includes network architectures, data encoding, data generation, and prompt construction process. As a first step, we construct a simple example with a one-layer transformer, and show an interesting result, namely when the pre-train data distribution is different from the query task distribution, a properly constructed context can shift the output distribution towards the query task distribution, in a quantifiable manner, leading to accurate prediction on the query topic. We then extend the findings in the previous step to a more general case, and derive the precise relationship between ICL performance, context length and the KL divergence between pre-train and query task distribution. Finally, we provide experiments to validate our theoretical results.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 36 references that are in the map. First posted on arXiv 26 Oct 2025.

Builds on

29 of 36 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2022In-context Learning and Induction Headscited by 964
  5. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  6. 2022Transformers learn in-context by gradient descentcited by 732
  7. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  8. 2019Release Strategies and the Social Impacts of Language Modelscited by 513
  9. 2021MetaICL: Learning to Learn In Contextcited by 454
  10. 2021Transformers Can Do Bayesian Inferencecited by 360
  11. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  12. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
17 more
  1. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 277
  2. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
  3. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  4. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 170
  5. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 159
  6. 2023Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learningcited by 149
  7. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 146
  8. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  9. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 118
  10. 2023In-Context Convergence of Transformerscited by 115
  11. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 96
  12. 2017Latent Dirichlet Allocation (LDA) and Topic modeling: models, applications, a surveycited by 91
  13. 2023How Do Transformers Learn Topic Structure: Towards a Mechanistic Understandingcited by 86
  14. 2023In-Context Learning through the Bayesian Prismcited by 83
  15. 2024Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimalitycited by 74
  16. 2024Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscapecited by 46
  17. 2022Transformers from an Optimization Perspectivecited by 27

Built on by

Not cited yet.