mlsanityData to 14 Oct 2026

Anti-Periodic Positional Encoding: M"obius Boundary Conditions Make In-Context Retrieval Reliable

arXiv 2607.21405 ↗First posted 23 Jul 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

M"obius RoPE is a rotary positional encoding built on the anti-periodic frequency ladder θi=π(2i+1)/N: every rotation plane advances by an odd multiple of π across the training context, so the positional holonomy is -1 and the two ends of the sequence are deterministically coupled through a closed-form Dirichlet "dipole"; to our knowledge this is the first anti-periodic boundary condition in positional encoding. We verify the theory numerically to ~ 10-6 and pretrain 48 models spanning six 160M-class and three 410M-class arms (2B FineWeb-Edu tokens each; the hybrid arm puts M"obius frequencies on 25% of heads). Hybrid perplexity is unchanged (29.66 vs. 29.72), but needle-in-a-haystack retrieval becomes reliable: 90.3±5.7% versus 63.3±31.4% at context 512 (n=6 seeds), observed worst seed 86% versus 14%, robust variance tests p=0.013-0.029 (unadjusted), recurring at 410M (Levene p=0.040). Matched controls isolate the mechanism: an aperiodic ladder in the same frequency band reproduces none of the effect, and a periodic (holonomy +1) ladder only a fraction. Swapping trained models' frequency table back to standard RoPE (weights frozen) collapses retrieval, with damage concentrated on far needles: trained models depend on this long-range geometry. A NoPE arm is even more reliable at short context but pays a 13% perplexity tax and extrapolates worst; only the anti-periodic hybrid pairs baseline perplexity with a high reliability floor. The effect is scoped to single-needle retrieval within the training window; a one-line frequency swap thus provides zero-cost insurance against the retrieval seed lottery.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 13 of its 18 references that are in the map. First posted on arXiv 23 Jul 2026.

Builds on

13 of 18 references.

References · 13

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  3. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  4. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  5. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  6. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  7. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  8. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
  9. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 213
  10. 2024Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalizationcited by 39
  11. 2024Expanding Expressivity in Transformer Models with M"obiusAttentioncited by 4
  12. 2026MrRoPE: Mixed-radix Rotary Position Embeddingcited by 4
1 more
  1. 2026Periodic RoPE for Infinite Context LLMscited by 2

Built on by

Not cited yet.