mlsanityData to 14 Oct 2026

Rotary Position Encodings for Graphs

arXiv 2509.22259 ↗First posted 26 Sep 2025cs.LG cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

We study the extent to which rotary position encodings (RoPE), a recent transformer position encoding algorithm broadly adopted in large language models (LLMs) and vision transformers (ViTs), can be applied to graph-structured data. We find that rotating tokens depending on the spectrum of the graph Laplacian efficiently injects structural information into the attention mechanism, boosting performance in synthetic and real-world graph learning tasks. This approach, coined _Wave-Induced Rotary Encodings_ (WIRE), enjoys intriguing theoretical properties: it recovers regular RoPE on grids, and depends asymptotically on the graph effective resistance. Unlike bias-based relative position encodings, WIRE is compatible with linear attention.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 41 of its 63 references that are in the map. First posted on arXiv 26 Sep 2025.

Builds on

41 of 63 references.

References · 41

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2017Graph Attention Networkscited by 8,923
  3. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2018How Powerful are Graph Neural Networks?cited by 4,393
  6. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  7. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
  8. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  9. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  10. 2020Rethinking Attention with Performerscited by 1,601
  11. 2018Self-Attention with Relative Position Representationscited by 1,120
  12. 2018Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networkscited by 881
29 more
  1. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  2. 2020PCT: Point cloud transformercited by 732
  3. 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 713
  4. 2020A Generalization of Transformer Networks to Graphscited by 661
  5. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  6. 2021Rethinking Graph Transformers with Spectral Attentioncited by 473
  7. 2019Provably Powerful Graph Networkscited by 389
  8. 2021Graph Neural Networks with Learnable Structural and Positional Representationscited by 333
  9. 2022Long Range Graph Benchmarkcited by 263
  10. 2022Sign and Basis Invariant Networks for Spectral Graph Representation Learningcited by 179
  11. 2023On Over-Squashing in Message Passing Neural Networks: The Impact of Width, Depth, and Topologycited by 166
  12. 2023Rethinking the Expressive Power of GNNs via Graph Biconnectivitycited by 119
  13. 2023Understanding Oversquashing in GNNs through the Lens of Effective Resistancecited by 97
  14. 2023Attending to Graph Transformerscited by 83
  15. 2022Expander Graph Propagationcited by 75
  16. 2020Learning to Encode Position for Transformer with Continuous Dynamical Modelcited by 61
  17. 2021SCENIC: A JAX Library for Computer Vision Research and Beyondcited by 57
  18. 2018Spectrally approximating large graphs with smaller graphscited by 56
  19. 2020A Large-Scale Database for Graph Representation Learningcited by 55
  20. 2021From block-Toeplitz matrices to differential equations on graphs: towards a general theory for scalable masked Transformerscited by 38
  21. 2021SHAPE: Shifted Absolute Position Embedding for Transformerscited by 37
  22. 2019Fast and Accurate Network Embeddings via Very Sparse Random Projectioncited by 29
  23. 2023Laplacian Canonization: A Minimalist Approach to Sign and Basis Invariant Spectral Embeddingcited by 21
  24. 2024Cayley Graph Propagationcited by 21
  25. 2025Learning the RoPEs: Better 2D and 3D Position Encodings with STRINGcited by 19
  26. 2022Affinity-Aware Graph Networkscited by 17
  27. 2024Linear Transformer Topological Masking with Graph Random Featurescited by 7
  28. 2024Masked Graph Transformer for Large-Scale Recommendationcited by 5
  29. 2023Multiresolution Graph Transformers and Wavelet Positional Encoding for Learning Hierarchical Structurescited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Analytic-Walk Rotary Positional Encodings for Graphscited by 0
  2. 2026Asymptotic Behavior of Transformerscited by 0