mlsanityData to 7 Oct 2026

An Information-Theoretic Analysis of In-Context Learning

arXiv 2401.15530 ↗First posted 28 Jan 2024cs.LG cs.IT math.ITcited by 39

Not on the map. Why?

Abstract (arXiv)

Previous theoretical results pertaining to meta-learning on sequences build on contrived assumptions and are somewhat convoluted. We introduce new information-theoretic tools that lead to an elegant and very general decomposition of error into three components: irreducible error, meta-learning error, and intra-task error. These tools unify analyses across many meta-learning challenges. To illustrate, we apply them to establish new results about in-context learning with transformers. Our theoretical results characterizes how error decays in both the number of training sequences and sequence lengths. Our results are very general; for example, they avoid contrived mixing time assumptions made by all prior results that establish decay of error with sequence length.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 31 of its 43 references that are in the map. First posted on arXiv 28 Jan 2024.

Builds on

31 of 43 references.

References · 31

  1. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  2. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  3. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  4. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  5. 2022Transformers learn in-context by gradient descentcited by 714
  6. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  7. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  8. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  9. 2021MetaICL: Learning to Learn In Contextcited by 450
  10. 2023Larger language models do in-context learning differentlycited by 386
  11. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  12. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
19 more
  1. 2022Transformers Learn Shortcuts to Automatacited by 260
  2. 2023Looped Transformers as Programmable Computerscited by 224
  3. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  4. 2015The Benefit of Multitask Representation Learningcited by 194
  5. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 168
  6. 2020Provable Meta-Learning of Linear Representationscited by 161
  7. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  8. 2023Representational Strengths and Limitations of Transformerscited by 132
  9. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 112
  10. 2023In-Context Convergence of Transformerscited by 112
  11. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 102
  12. 2023Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformercited by 101
  13. 2022Vision Transformers provably learn spatial structurecited by 92
  14. 2023How Do Transformers Learn Topic Structure: Towards a Mechanistic Understandingcited by 85
  15. 2023Transformers as Support Vector Machinescited by 67
  16. 2023JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attentioncited by 52
  17. 2018Incremental Learning-to-Learn with Statistical Guaranteescited by 16
  18. 2023Uncovering hidden geometry in Transformers via disentangling position and contextcited by 15
  19. 2023Metalearning with Very Few Samples Per Taskcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 15; 2026: 9.

39 papers.

Sort:

By area:

  1. 2024Many-Shot In-Context Learningcited by 203
  2. 2024Can large language models explore in-context?cited by 75
  3. 2024Dual Operating Modes of In-Context Learningcited by 42
  4. 2024In-Context Learning with Representations: Contextual Generalization of Trained Transformerscited by 36
  5. 2024Mechanics of Next Token Prediction with Self-Attentioncited by 34
  6. 2024Large Language Models as Markov Chainscited by 33
  7. 2024Pretrained transformer efficiently learns low-dimensional target functions in-contextcited by 29
  8. 2025TACO: Enhancing Multimodal In-context Learning via Task Mapping-Guided Sequence Configurationcited by 23
  9. 2024Estimating the Hallucination Rate of Generative AIcited by 20
  10. 2025Softmax is 1/2-Lipschitz: A tight bound across all ℓp normscited by 18
  11. 2025In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learningcited by 17
  12. 2024Revisiting In-context Learning Inference Circuit in Large Language Modelscited by 16
27 more
  1. 2025HyperTree Planning: Enhancing LLM Reasoning via Hierarchical Thinkingcited by 14
  2. 2024A Survey to Recent Progress Towards Understanding In-Context Learningcited by 12
  3. 2025Gating is Weighting: Understanding Gated Linear Attention through In-context Learningcited by 11
  4. 2024Optimizing Attention with Mirror Descent: Generalized Max-Margin Token Selectioncited by 9
  5. 2025Federated In-Context Learning: Iterative Refinement for Improved Answer Qualitycited by 7
  6. 2024Understanding the Training and Generalization of Pretrained Transformer for Sequential Decision Makingcited by 4
  7. 2025Provable In-Context Learning of Nonlinear Regression with Transformerscited by 4
  8. 2024From Unstructured Data to In-Context Learning: Exploring What Tasks Can Be Learned and Whencited by 3
  9. 2024Towards Better Understanding of In-Context Learning Ability from In-Context Uncertainty Quantificationcited by 2
  10. 2024Towards the Effect of Examples on In-Context Learning: A Theoretical Case Studycited by 2
  11. 2025StaICC: Standardized Evaluation for Classification Task in In-context Learningcited by 2
  12. 2025An Auditable Agent Platform For Automated Molecular Optimisationcited by 2
  13. 2025Stability of In-Context Learning: A Spectral Coverage Perspectivecited by 2
  14. 2026Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Modelscited by 2
  15. 2025Brewing Knowledge in Context: Distillation Perspectives on In-Context Learningcited by 1
  16. 2025Graph of Agents: Principled Long Context Modeling by Emergent Multi-Agent Collaborationcited by 1
  17. 2025Optimal In-context Adaptivity and Distributional Robustness of Transformerscited by 1
  18. 2026Bayesian Optimality of In-Context Learning with Selective State Spacescited by 1
  19. 2026Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottleneckscited by 1
  20. 2025Technical Debt in In-Context Learning: Diminishing Efficiency in Long Contextcited by 0
  21. 2025Few-Shot and Training-Free Review Generation via Conversational Promptingcited by 0
  22. 2026Meta-Sel: Efficient Demonstration Selection for In-Context Learning via Supervised Meta-Learningcited by 0
  23. 2026Quantum Maximum Likelihood Prediction via Hilbert Space Embeddingscited by 0
  24. 2026Identifiability and Order-Dimension Limits of In-Context Learning on Partial Orderscited by 0
  25. 2026From Fixed Keys to Readable Schemas: Small Language Models for Vehicle Agent Function Callscited by 0
  26. 2026On the Capability and Limitation of Hard Promptcited by 0
  27. 2026Theory on Attention Dynamics for Out-of-Distribution In-Context Learningcited by 0