mlsanityData to 7 Oct 2026

Kinetic theory for Transformers and the lost-in-the-middle phenomenon

arXiv 2605.09213 ↗First posted 9 May 2026math.AP cs.LG math.PRcited by 0

Not on the map. Why?

Abstract (arXiv)

We study causal self-attention dynamics -- a toy model for decoder Transformers -- which we interpret as a non-exchangeable interacting particle system. Adapting cumulant expansions to the triangular causal dependency structure of the model, and appealing to non-hierarchical methods to estimate correlations using Glauber calculus, we prove a quantitative mean-field limit result and a next-order characterization of correlations. For iid uniformly distributed tokens, the limiting correlation equation can be solved in closed form and we obtain a rigorous explanation of the empirically observed lost-in-the-middle phenomenon: the token retrieval profile, as a function of the source position in the prompt, is mathsfU-shaped, with primacy, recency, and a unique interior minimum under an explicit smallness condition.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 26 of its 39 references that are in the map. First posted on arXiv 9 May 2026.

Builds on

26 of 39 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  5. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  6. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  7. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 200
  8. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 136
  9. 2023The emergence of clusters in self-attention dynamicscited by 133
  10. 2025Why do LLMs attend to the first token?cited by 122
  11. 2024Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilizationcited by 97
  12. 2025On the Emergence of Position Bias in Transformerscited by 57
14 more
  1. 2025Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coincited by 47
  2. 2024Dynamic metastability in the self-attention modelcited by 42
  3. 2024Emergence of meta-stable clustering in mean-field transformer modelscited by 35
  4. 2024Clustering in Causal Attention Maskingcited by 31
  5. 2025A Unified Perspective on the Dynamics of Deep Transformerscited by 30
  6. 2024Measure-to-measure interpolation using Transformerscited by 28
  7. 2025A multiscale analysis of mean-field transformers in the moderate interaction regimecited by 24
  8. 2025Quantitative Clustering in Mean-Field Transformer Modelscited by 23
  9. 2025Synchronization of mean-field models on the circlecited by 10
  10. 2026Perceptrons and localization of attention's mean-field landscapecited by 7
  11. 2026Clustering in Deep Stochastic Transformerscited by 5
  12. 2026A Structural Theory of Position Bias in Transformerscited by 3
  13. 2026Lost in the Middle at Birth: An Exact Theory of Transformer Position Biascited by 3
  14. 2026Homogenized Transformerscited by 3

Built on by

Not cited yet.