In-context classification: Calibration and noise in linear attention with softmax readout
Not on the map. Why?
Abstract (arXiv)
We study in-context binary classification in a simple attention model trained to minimise a logistic loss. Each prompt has a latent logistic teacher, and the learner must infer its direction from Bernoulli-labeled context examples. We derive an asymptotic prediction for the test logit in a high-dimensional regime with context length proportional to token dimension, and total training prompts proportional to dimension squared. Our formula cleanly separates calibration, test-time context noise, and non-isotropic parameter-estimation noise as interpretable effects on in-context performance. Our analysis also identifies the ridgeless separability regime and addresses the effects of context length, pretraining-set size, regularisation, and sampling temperature on in-context classification performance. The theory's predictions closely agree with finite-dimensional simulations and fully trained linear-attention models, while experiments with standard softmax attention show qualitatively related behaviour beyond the regime assumed by our analysis.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 29 references that are in the map. First posted on arXiv 12 Oct 2026.
Builds on
23 of 29 references.
References · 23
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2022Emergent Abilities of Large Language Modelscited by 2,512
- 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
- 2022Transformers learn in-context by gradient descentcited by 732
- 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
- 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
- 2023Trained Transformers Learn Linear Models In-Contextcited by 337
- 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
- 2020Generalisation error in learning with random features and the hidden manifold modelcited by 127
- 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 118
- 2017Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Modelscited by 116
- 2023Linear attention is (maybe) all you need (to understand transformer optimization)cited by 98
11 more
- 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 96
- 2018A modern maximum-likelihood theory for high-dimensional logistic regressioncited by 81
- 2024Asymptotic theory of in-context learning by linear attentioncited by 55
- 2018The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regressioncited by 46
- 2024Pretrained transformer efficiently learns low-dimensional target functions in-contextcited by 31
- 2023Are Gaussian data all you need? Extents and limits of universality in high-dimensional generalized linear estimationcited by 26
- 2025Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Timecited by 16
- 2024Transformers Learn Low Sensitivity Functions: Investigations and Implicationscited by 14
- 2025Pretrain-Test Task Alignment Governs Generalization in In-Context Learningcited by 8
- 2026Transformers Efficiently Perform In-Context Logistic Regression via Normalized Gradient Descentcited by 1
- 2026Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatchcited by 1
Built on by
Not cited yet.