mlsanityData to 14 Oct 2026

Positional Encoding via Token-Aware Phase Attention

arXiv 2509.12635 ↗First posted 16 Sep 2025cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

We prove under practical assumptions that Rotary Positional Embedding (RoPE) introduces an intrinsic distance-dependent bias in attention scores that limits RoPE's ability to model long-context. RoPE extension methods may alleviate this issue, but they typically require post-hoc adjustments after pretraining, such as rescaling or hyperparameters retuning. This paper introduces Token-Aware Phase Attention (TAPA), a new positional encoding method that incorporates a learnable phase function into the attention mechanism. TAPA preserves token interactions over long range, extends to longer contexts with direct and light continual pretraining, extrapolates to unseen lengths, and attains substantially lower perplexity and stronger retrieval performance in the long-context regime than RoPE-style baselines.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 34 of its 48 references that are in the map. First posted on arXiv 16 Sep 2025.

Builds on

34 of 48 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  8. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  10. 2017Decoupled Weight Decay Regularizationcited by 1,831
  11. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  12. 2018Self-Attention with Relative Position Representationscited by 1,120
22 more
  1. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  2. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  3. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  4. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  5. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  6. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  7. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 393
  8. 2021FNet: Mixing Tokens with Fourier Transformscited by 347
  9. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  10. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
  11. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  12. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 204
  13. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  14. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 150
  15. 2022A Length-Extrapolatable Transformercited by 127
  16. 2023Scaling Laws of RoPE-based Extrapolationcited by 125
  17. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 124
  18. 2020Improve Transformer Models with Better Relative Position Embeddingscited by 73
  19. 2021CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddingscited by 47
  20. 2024Length Generalization of Causal Transformers without Position Encodingcited by 47
  21. 2023Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddingscited by 23
  22. 2024Mesa-Extrapolation: A Weave Position Encoding Method for Enhanced Extrapolation in LLMscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Parabolic Position Encoding: Vision-Centric, Principled, Extrapolatable, Generalcited by 0