mlsanityData to 7 Oct 2026

Transformers perform adaptive partial pooling

arXiv 2602.03980 ↗First posted 3 Feb 2026cs.CL cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Any language model must decide what to say in novel contexts based on information from similar contexts. But what about contexts that are not novel but merely infrequent? In hierarchical regression, the model's predictions for behavior in a context are affected by observations from similar contexts to the extent that 1) the current context is infrequent and 2) different contexts behave similarly. This is called adaptive partial pooling. This paper shows that next-word predictions of a transformer (GPT2) are affected by observations from outside the current context, but this pooling reduces with more training. Pooling is affected by context frequency, context number (type frequency) and context variability in a qualitatively similar way to hierarchical regression. However, there is a "sweet spot" in training at which the transformer best matches the behavior of hierarchical regression. This is the point at which the effect of context frequency on pooling is at maximum.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 9 of its 35 references that are in the map. First posted on arXiv 3 Feb 2026.

Builds on

9 of 35 references.

References · 9

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  4. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 452
  5. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 163
  6. 2019One Epoch Is All You Needcited by 41
  7. 2024Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Timescited by 37
  8. 2025Language Models Grow Less Humanlike beyond Phase Transitioncited by 10
  9. 2023Frequency effects in Linear Discriminative Learningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Exemplars in Disguise: Pure Exemplar Models Mimic Abstraction-First Learningcited by 0
  2. 2026Generalization and Memorization along the Learning Trajectory of Neural Language Models: A Geometric Account of Categorizationcited by 0
  3. 2026Neural Language Models Learn the Contextual Distributions of Dependency Structures: a statistical learning theory to compositionalitycited by 0