mlsanityData to 14 Oct 2026

Analyzing limits for in-context learning

arXiv 2502.03503 ↗First posted 5 Feb 2025stat.ML cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Our paper challenges claims from prior research that transformer-based models, when learning in context, implicitly implement standard learning algorithms. We present empirical evidence inconsistent with this view and provide a mathematical analysis demonstrating that transformers cannot achieve general predictive accuracy due to inherent architectural limitations.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 24 of its 33 references that are in the map. First posted on arXiv 5 Feb 2025.

Builds on

24 of 33 references.

References · 24

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2016Layer Normalizationcited by 5,669
  4. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  5. 2022In-context Learning and Induction Headscited by 964
  6. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  7. 2022Transformers learn in-context by gradient descentcited by 732
  8. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  9. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  10. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  11. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  12. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 246
12 more
  1. 2023Birth of a Transformer: A Memory Viewpointcited by 161
  2. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 146
  3. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 118
  4. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 96
  5. 2023In-Context Learning through the Bayesian Prismcited by 83
  6. 2023Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functionscited by 63
  7. 2023Characterizing Mechanisms for Factual Recall in Language Modelscited by 48
  8. 2023Normalization Layers Are All That Sharpness-Aware Minimization Needscited by 27
  9. 2023The Crucial Role of Normalization in Sharpness-Aware Minimizationcited by 26
  10. 2024How Well Can Transformers Emulate In-context Newton's Method?cited by 25
  11. 2022How to Dissect a Muppet: The Structure of Transformer Embedding Spacescited by 20
  12. 2024Re-examining learning linear functions in contextcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 1.

6 papers.

  1. 2025Transformers are Inherently Succinctcited by 4
  2. 2025Test time training enhances in-context learning of nonlinear functionscited by 3
  3. 2026Understanding In-Context Learning for Nonlinear Regression with Transformers: Attention as Featurizercited by 3
  4. 2025TELL-TALE: Task Efficient LLMs with Task Aware Layer Eliminationcited by 2
  5. 2025SSA: Improving Performance With a Better Scoring Functioncited by 1
  6. 2025COCORELI: Enforcing Execution Preconditions for Reliable Collaborative Instruction Followingcited by 0