mlsanityData to 14 Oct 2026

Deconstructing Positional Information: From Attention Logits to Training Biases

arXiv 2505.13027 ↗First posted 19 May 2025cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Positional encodings enable Transformers to incorporate sequential information, yet their theoretical understanding remains limited to two properties: distance attenuation and translation invariance. Because natural language lacks purely positional data, the interplay between positional and semantic information is still underexplored. We address this gap by deconstructing the attention-logit computation and providing a structured analysis of positional encodings, categorizing them into additive and multiplicative forms. The differing properties of these forms lead to distinct mechanisms for capturing positional information. To probe this difference, we design a synthetic task that explicitly requires strong integration of positional and semantic cues. As predicted, multiplicative encodings achieve a clear performance advantage on this task. Moreover, our evaluation reveals a hidden training bias: an information aggregation effect in shallow layers that we term the single-head deposit pattern. Through ablation studies and theoretical analysis, we proved that this phenomenon is inherent in multiplicative encodings. These findings deepen the understanding of positional encodings and call for further study of their training dynamics.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 21 of its 32 references that are in the map. First posted on arXiv 19 May 2025.

Builds on

21 of 32 references.

References · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2018Self-Attention with Relative Position Representationscited by 1,120
  6. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  7. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  8. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  9. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
  10. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 244
  11. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 150
  12. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 115
9 more
  1. 2022KERPLE: Kernelized Relative Positional Embedding for Length Extrapolationcited by 86
  2. 2024Length Generalization of Causal Transformers without Position Encodingcited by 47
  3. 2025Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understandingcited by 47
  4. 2023CLEX: Continuous Length Extrapolation for Large Language Modelscited by 44
  5. 2022Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysiscited by 42
  6. 2024Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalizationcited by 39
  7. 2021SHAPE: Shifted Absolute Position Embedding for Transformerscited by 37
  8. 2025DRoPE: Directional Rotary Position Embedding for Efficient Agent Interaction Modelingcited by 9
  9. 2024Position Information Emerges in Causal Transformers Without Positional Encodings via Similarity of Nearby Embeddingscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Phase Structure in Rotary Attention: A Spectral Framework for Semantic Continuity and Execution-Boundary Governancecited by 1
  2. 2026Three-Phase Transformercited by 0
  3. 2026Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoderscited by 0
  4. 2026Self-Attention Dynamics with Rotary Position Embeddings: Twisted States and Explicit Consensus Rates on the Spherecited by 0
  5. 2026Dynamic Positional Attention Modulation for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 0