mlsanityData to 14 Oct 2026

Re-examining learning linear functions in context

arXiv 2411.11465 ↗First posted 18 Nov 2024cs.LG cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

In-context learning (ICL) has emerged as a powerful paradigm for easily adapting Large Language Models (LLMs) to various tasks. However, our understanding of how ICL works remains limited. We explore a simple model of ICL in a controlled setup with synthetic training data to investigate ICL of univariate linear functions. We experiment with a range of GPT-2-like transformer models trained from scratch. Our findings challenge the prevailing narrative that transformers adopt algorithmic approaches like linear regression to learn a linear function in-context. These models fail to generalize beyond their training distribution, highlighting fundamental limitations in their capacity to infer abstract task structures. Our experiments lead us to propose a mathematically precise hypothesis of what the model might be learning.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 25 of its 34 references that are in the map. First posted on arXiv 18 Nov 2024.

Builds on

25 of 34 references.

References · 25

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  4. 2022In-context Learning and Induction Headscited by 964
  5. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  6. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  7. 2022Transformers learn in-context by gradient descentcited by 732
  8. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  9. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  10. 2017Exploring Generalization in Deep Learningcited by 600
  11. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 550
  12. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
13 more
  1. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  2. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  3. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 170
  4. 2023LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Proverscited by 168
  5. 2017Generalization in Deep Learningcited by 146
  6. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 146
  7. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 118
  8. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 96
  9. 2023In-Context Learning through the Bayesian Prismcited by 83
  10. 2023Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functionscited by 63
  11. 2024How Well Can Transformers Emulate In-context Newton's Method?cited by 25
  12. 2013On Learnability, Complexity and Stabilitycited by 4
  13. 2024On Explaining with Attention Matricescited by 2

Built on by

Cited by year

Citing papers per year: 2025: 5.

5 papers.

  1. 2025Analyzing limits for in-context learningcited by 6
  2. 2025TELL-TALE: Task Efficient LLMs with Task Aware Layer Eliminationcited by 2
  3. 2025SSA: Improving Performance With a Better Scoring Functioncited by 1
  4. 2025Genomic Next-Token Predictors are In-Context Learnerscited by 1
  5. 2025COCORELI: Enforcing Execution Preconditions for Reliable Collaborative Instruction Followingcited by 0