mlsanityData to 14 Oct 2026

Learning In-context n-grams with Transformers: Sub-n-grams Are Near-stationary Points

arXiv 2508.12837 ↗First posted 18 Aug 2025cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

Motivated by empirical observations of prolonged plateaus and stage-wise progression during training, we investigate the loss landscape of transformer models trained on in-context next-token prediction tasks. In particular, we focus on learning in-context n-gram language models under cross-entropy loss, and establish a sufficient condition for parameter configurations to be stationary points. We then construct a set of parameter configurations for a simplified transformer model that represent k-gram estimators (for k ≤ n), and show that the gradient of the population loss at these solutions vanishes in the limit of infinite sequence length and parameter norm. This reveals a key property of the loss landscape: sub-n-grams are near-stationary points of the population cross-entropy loss, offering theoretical insight into widely observed phenomena such as stage-wise learning dynamics and emergent phase transitions. These insights are further supported by numerical experiments that illustrate the learning dynamics of n-grams, characterized by discrete transitions between near-stationary solutions.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 35 of its 48 references that are in the map. First posted on arXiv 18 Aug 2025.

Builds on

35 of 48 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Emergent Abilities of Large Language Modelscited by 2,512
  4. 2022In-context Learning and Induction Headscited by 964
  5. 2022Transformers learn in-context by gradient descentcited by 732
  6. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  7. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  8. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  9. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  10. 2018A mathematical theory of semantic development in deep neural networkscited by 186
  11. 2023Birth of a Transformer: A Memory Viewpointcited by 161
  12. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 150
23 more
  1. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 143
  2. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 124
  3. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 124
  4. 2024The Evolution of Statistical Induction Heads: In-Context Learning Markov Chainscited by 118
  5. 2024How Transformers Learn Causal Structure with Gradient Descentcited by 118
  6. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 85
  7. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 76
  8. 2019The Implicit Bias of Depth: How Incremental Learning Drives Generalizationcited by 64
  9. 2023Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functionscited by 63
  10. 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 57
  11. 2023Transformers learn through gradual rank increasecited by 52
  12. 2024Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformerscited by 50
  13. 2021Implicit Regularization in Tensor Factorizationcited by 44
  14. 2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 42
  15. 2024Attention with Markov: A Framework for Principled Analysis of Transformers via Markov Chainscited by 40
  16. 2024Large Language Models as Markov Chainscited by 35
  17. 2022Algorithmic Regularization in Model-free Overparametrized Asymmetric Matrix Factorizationcited by 34
  18. 2024Transformers Can Represent n-gram Language Modelscited by 32
  19. 2022Incremental Learning in Diagonal Linear Networkscited by 28
  20. 2024Transformers on Markov Data: Constant Depth Sufficescited by 27
  21. 2024Understanding Transformers via N-gram Statisticscited by 26
  22. 2024Task Diversity Shortens the ICL Plateaucited by 5
  23. 2024A Mechanistic Study of Transformers Training Dynamicscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 14.

16 papers.

Sort:

By area:

  1. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 15
  2. 2026To Use or not to Use Muon: How Simplicity Bias in Optimizers Matterscited by 10
  3. 2026Length Generalization Bounds for Transformerscited by 4
  4. 2025Softmax as Linear Attention in the Large-Prompt Regime: a Measure-based Perspectivecited by 3
  5. 2026Incremental Learning of Sparse Attention Patterns in Transformerscited by 2
  6. 2026Gradient Flow Polarizes Softmax Outputs towards Low-Entropy Solutionscited by 2
  7. 2026Incremental Learning in Mirror Flowscited by 2
  8. 2026Invariant Learning Dynamics of Transformers in Inductive Reasoning Taskscited by 2
  9. 2026Transformers Learn Latent Mixture Models In-Context via Mirror Descentcited by 1
  10. 2026Smooth Scaling Laws Hide Stepwise Token Learningcited by 1
  11. 2026Induction Heads Interpolate N-Gramscited by 1
  12. 2026Gaussian Match-and-Copy: A Minimalist Benchmark for Studying Transformer Inductioncited by 0
4 more
  1. 2026Task-Centric Acceleration of Small-Language Modelscited by 0
  2. 2026A Theory of Saddle Escape in Deep Nonlinear Networkscited by 0
  3. 2026Extracting Algorithms in Pre-trained LLMs: A Case on Hidden Markov Modelscited by 0
  4. 2026Length Generalization for Transformers via Compressioncited by 0