A Structural Theory of Position Bias in Transformers
Not on the map. Why?
Abstract (arXiv)
Transformer models systematically favor certain token positions, yet the architectural origins of this position bias remain poorly understood. This bias is closely connected to the Lost-in-the-Middle phenomenon, where models underutilize information placed in the middle of the context. We show that Lost-in-the-Middle-type behavior can arise from the architecture of causal Transformers itself. To do so, we develop a structural theory of position bias based on residual-aware cumulative attention rollout. At finite depth, causal masking and residual connections induce broad, often U-shaped, influence profiles. At infinite depth, our framework resolves a discrepancy between prior attention-only collapse theory and practical Transformer behavior: residual connections fundamentally change cumulative attention dynamics. Empirically, the predicted profiles closely match measured input-token influence in pretrained language models.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 32 of its 34 references that are in the map. First posted on arXiv 18 Feb 2026.
Builds on
32 of 34 references.
References · 32
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
- 2018Neural Ordinary Differential Equationscited by 3,733
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
- 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
- 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
- 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
- 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
- 2020Quantifying Attention Flow in Transformerscited by 759
- 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
20 more
- 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
- 2017Stable Architectures for Deep Neural Networkscited by 340
- 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
- 2018Deep Neural Networks Motivated by Partial Differential Equationscited by 187
- 2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92
- 2024Transformers need glasses! Information over-squashing in language taskscited by 87
- 2024Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encodingcited by 72
- 2025On the Emergence of Position Bias in Transformerscited by 57
- 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMscited by 54
- 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 46
- 2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 44
- 2024Serial Position Effects of Large Language Modelscited by 30
- 2024StableMask: Refining Causal Masking in Decoder-only Transformercited by 20
- 2024From Attention to Activation: Unravelling the Enigmas of Large Language Modelscited by 13
- 2025Where to show Demos in Your Prompt: A Positional Bias of In-Context Learningcited by 13
- 2025Positional Biases Shift as Inputs Approach Context Window Limitscited by 9
- 2025Lost in the Middle: An Emergent Property from Information Retrieval Demands in LLMscited by 8
- 2025Beyond Early-Token Bias: Model-Specific and Language-Specific Position Effects in Multilingual LLMscited by 5
- 2026Jacobian Scopes: token-level causal attributions in LLMscited by 4
- 2026Layer-wise Positional Bias in Short-Context Language Modelingcited by 1
Built on by
Citing papers per year: 2026: 3.
3 papers.