mlsanityData to 7 Oct 2026

Data Distributional Properties Drive Emergent In-Context Learning in Transformers

arXiv 2205.05055 ↗First posted 22 Apr 2022cs.LG cs.AI cs.CLcited by 303

Not on the map. Why?

Abstract (arXiv)

Large transformer-based models are able to perform in-context few-shot learning, without being explicitly trained for it. This observation raises the question: what aspects of the training regime lead to this emergent behavior? Here, we show that this behavior is driven by the distributions of the training data itself. In-context learning emerges when the training data exhibits particular distributional properties such as burstiness (items appear in clusters rather than being uniformly distributed over time) and having large numbers of rarely occurring classes. In-context learning also emerges more strongly when item meanings or interpretations are dynamic rather than fixed. These properties are exemplified by natural language, but are also inherent to naturalistic data in a wide range of other domains. They also depart significantly from the uniform, i.i.d. training distributions typically used for standard supervised learning. In our initial experiments, we found that in-context learning traded off against more conventional weight-based learning, and models were unable to achieve both simultaneously. However, our later experiments uncovered that the two modes of learning could co-exist in a single model when it was trained on data following a skewed Zipfian distribution -- another common property of naturalistic data, including language. In further experiments, we found that naturalistic data distributions were only able to elicit in-context learning in transformers, and not in recurrent models. In sum, our findings indicate how the transformer architecture works together with particular properties of the training data to drive the intriguing emergent in-context learning behaviour of large language models, and how future work might encourage both in-context and in-weights learning in domains beyond language.

Key facts. Cited by 303 papers in the map (data through 7 Oct 2026). Builds on 18 of its 42 references that are in the map. First posted on arXiv 22 Apr 2022.

Builds on

18 of 42 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  5. 2016Matching Networks for One Shot Learningcited by 2,246
  6. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  7. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  8. 2020Hopfield Networks is All You Needcited by 461
  9. 2016Learning to reinforcement learncited by 369
  10. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  11. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 323
  12. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
6 more
  1. 2020Large Associative Memory Problem in Neurobiology and Machine Learningcited by 110
  2. 2022Can Wikipedia Help Offline Reinforcement Learning?cited by 76
  3. 2019The Omniglot challenge: a 3-year progress reportcited by 54
  4. 2020Grounded Language Learning Fast and Slowcited by 54
  5. 2022Zipfian environments for Reinforcement Learningcited by 13
  6. 2013Burstiness and spreading on temporal networkscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 14; 2023: 88; 2024: 71; 2025: 82; 2026: 48.

303 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2022A Survey on In-context Learningcited by 717
  3. 2022Transformers learn in-context by gradient descentcited by 714
  4. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  5. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  6. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  7. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 319
  8. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 293
  9. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  10. 2022Emergent Analogical Reasoning in Large Language Modelscited by 239
  11. 2023Large Language Models as General Pattern Machinescited by 196
  12. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
38 more
  1. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 175
  2. 2022Talking About Large Language Modelscited by 174
  3. 2023Birth of a Transformer: A Memory Viewpointcited by 154
  4. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 149
  5. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 138
  6. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 138
  7. 2023The Quantization Model of Neural Scalingcited by 137
  8. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
  9. 2022Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modelingcited by 126
  10. 2023The Learnability of In-Context Learningcited by 126
  11. 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 115
  12. 2024The Evolution of Statistical Induction Heads: In-Context Learning Markov Chainscited by 115
  13. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  14. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  15. 2023Are Emergent Abilities in Large Language Models just In-Context Learning?cited by 109
  16. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 104
  17. 2023Why think step by step? Reasoning emerges from the locality of experiencecited by 103
  18. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 102
  19. 2023Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecastingcited by 102
  20. 2024Not All Layers of LLMs Are Necessary During Inferencecited by 95
  21. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  22. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
  23. 2024In-Context Language Learning: Architectures and Algorithmscited by 86
  24. 2023Cognitive Mirage: A Review of Hallucinations in Large Language Modelscited by 83
  25. 2024Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Datacited by 83
  26. 2023In-Context Learning through the Bayesian Prismcited by 78
  27. 2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
  28. 2023How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representationscited by 72
  29. 2023The Transient Nature of Emergent In-Context Learning in Transformerscited by 72
  30. 2023TinyGSM: achieving >80% on GSM8k with small language modelscited by 68
  31. 2023Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functionscited by 62
  32. 2023Towards Better Chain-of-Thought Prompting Strategies: A Surveycited by 62
  33. 2023Gen2Sim: Scaling up Robot Learning in Simulation with Generative Modelscited by 62
  34. 2023Meta-in-context learning in large language modelscited by 59
  35. 2024Asymptotic theory of in-context learning by linear attentioncited by 54
  36. 2023Large Language Model Enhanced Multi-Agent Systems for 6G Communicationscited by 53
  37. 2022Transformers generalize differently from information stored in context vs in weightscited by 49
  38. 2023In-Context Learning Learns Label Relationships but Is Not Conventional Learningcited by 48

50 most cited of 303 All 303 (JSON)