mlsanityData to 14 Oct 2026

PJ-RoPE: A Fourier-Jet-Affine Position Space for Relative Attention

arXiv 2606.05345 ↗First posted 3 Jun 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We organize relative-position mechanisms in attention as a learnable Fourier-Jet-Affine position space. The starting point is lag-shift dynamics: a relative-position kernel is a response function of the lag (d=i-j), and the one-step shift ((Ef)(d)=f(d+1)) gives a compact classification of finite structured responses through constant-coefficient difference modules. In this view, RoPE supplies simple Fourier roots, Jordan-RoPE thickens these roots into finite Fourier jets, and ALiBi supplies the repeated unit-root affine direction. NTK-aware RoPE scaling fits the same structure as a spectral flow of simple Fourier roots: moving the frequency grid generates first Fourier-jet tangent directions, while higher Taylor directions generate higher jets. PJ-RoPE makes these jet directions explicit and learnable, and uses the resulting space to measure task-level sector selection. The framework separates scalar PJ-bias kernels from exact PJ-rotary feature transforms, introduces sector-gate, effective-mass, functional-energy, and leave-one-order-out diagnostics, and stabilizes high-order coordinates with LC/rapidity compactification. Controlled probes recover designed sectors; synthetic teachers show trainable use; small byte-level language runs favor NTK-aware RoPE plus affine recency; symbolic music-token streams keep LC/affine variants strong with measurable high-order corrections; and LC diagnostics quantify the stability-resolution tradeoff.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 25 of its 30 references that are in the map. First posted on arXiv 3 Jun 2026.

Builds on

25 of 30 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  5. 2018Self-Attention with Relative Position Representationscited by 1,120
  6. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  7. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  8. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  9. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  10. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
  11. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  12. 2018Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Datasetcited by 217
13 more
  1. 2018Music Transformercited by 211
  2. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 150
  3. 2022A Length-Extrapolatable Transformercited by 127
  4. 2023Scaling Laws of RoPE-based Extrapolationcited by 125
  5. 2022KERPLE: Kernelized Relative Positional Embedding for Length Extrapolationcited by 86
  6. 2016Learning Features of Music from Scratchcited by 58
  7. 2024LieRE: Lie Rotational Positional Encodingscited by 18
  8. 2023Algebraic Positional Encodingscited by 10
  9. 2025Group Representational Position Encodingcited by 5
  10. 2024MEP: Multiple Kernel Learning Enhancing Relative Positional Encoding Length Extrapolationcited by 3
  11. 2023HyPE: Attention with Hyperbolic Biases for Relative Positional Encodingcited by 2
  12. 2022FastRPB: a Scalable Relative Positional Encoding for Long Sequence Taskscited by 1
  13. 2026Jordan-RoPE: Non-Semisimple Relative Positional Encoding via Complex Jordan Blockscited by 1

Built on by

Not cited yet.