mlsanityData to 14 Oct 2026

Group Representational Position Encoding

arXiv 2512.07805 ↗First posted 8 Dec 2025cs.LG cs.AI cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

We present GRAPE (Group Representational Position Encoding), a unified framework for positional encoding based on group actions. GRAPE unifies two families of mechanisms: (i) multiplicative rotations (Multiplicative GRAPE) in operatornameSO(d) and (ii) additive logit biases (Additive GRAPE) arising from unipotent actions in the general linear group GL. In Multiplicative GRAPE, a position n ∈ mathbbZ (or t ∈ ℝ) acts as G(n) = exp(n , ω , L) with a rank-2 skew-symmetric generator L ∈ ℝd × d, yielding a relative, compositional, norm-preserving map with a closed-form matrix exponential. RoPE is recovered exactly when the d/2 planes correspond to canonical coordinate pairs with a log-uniform spectrum. Learned commuting subspaces and compact non-commuting mixtures strictly extend this geometry to capture cross-subspace feature coupling at O(d) and O(r d) cost per head, respectively. In Additive GRAPE, additive logits arise from rank-1 (or low-rank) unipotent actions, recovering ALiBi and the Forgetting Transformer (FoX) as exact special cases while preserving an exact relative law and streaming cacheability. Overall, GRAPE provides a principled design space for positional geometry in long-context models, subsuming RoPE and ALiBi as special cases. Project page: https://github.com/model-architectures/GRAPE.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 45 of its 50 references that are in the map. First posted on arXiv 8 Dec 2025.

Builds on

45 of 50 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2025Qwen3 Technical Reportcited by 8,820
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  8. 2020Longformer: The Long-Document Transformercited by 3,413
  9. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  12. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
33 more
  1. 2020Rethinking Attention with Performerscited by 1,601
  2. 2018Self-Attention with Relative Position Representationscited by 1,120
  3. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  4. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  5. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  6. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  7. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  8. 2024Rotary Position Embedding for Vision Transformercited by 236
  9. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  10. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 150
  11. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 136
  12. 2022A Length-Extrapolatable Transformercited by 127
  13. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 117
  14. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 115
  15. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 111
  16. 2022KERPLE: Kernelized Relative Positional Embedding for Length Extrapolationcited by 86
  17. 2020Learning to Encode Position for Transformer with Continuous Dynamical Modelcited by 61
  18. 2024Contextual Position Encoding: Learning to Count What's Importantcited by 54
  19. 2019Encoding word order in complex embeddingscited by 53
  20. 2025Forgetting Transformer: Softmax Attention with a Forget Gatecited by 53
  21. 2021CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddingscited by 47
  22. 2024Length Generalization of Causal Transformers without Position Encodingcited by 47
  23. 2022Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysiscited by 42
  24. 2021SHAPE: Shifted Absolute Position Embedding for Transformerscited by 37
  25. 2025PaTH Attention: Position Encoding via Accumulating Householder Transformationscited by 34
  26. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 31
  27. 2020DA-Transformer: Distance-aware Transformercited by 23
  28. 2021The Case for Translation-Invariant Self-Attention in Transformer-Based Language Modelscited by 23
  29. 2025Learning the RoPEs: Better 2D and 3D Position Encodings with STRINGcited by 19
  30. 2024LieRE: Lie Rotational Positional Encodingscited by 18
  31. 2022The Curious Case of Absolute Position Embeddingscited by 17
  32. 2023Linearized Relative Positional Encodingcited by 17
  33. 2023Algebraic Positional Encodingscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Jordan-RoPE: Non-Semisimple Relative Positional Encoding via Complex Jordan Blockscited by 1
  2. 2026PJ-RoPE: A Fourier-Jet-Affine Position Space for Relative Attentioncited by 0
  3. 2026The Token Is a Group Element: On Lie-Algebra Attention over Matrix Lie Groupscited by 0
  4. 2026MinkowskiPE: Minkowski Positional Encoding for Spatiotemporal Perceptioncited by 0
  5. 2026How Local Mixing Encodes Relative Position in Global NoPE Attentioncited by 0