mlsanityData to 7 Oct 2026

LieRE: Lie Rotational Positional Encodings

arXiv 2406.10322 ↗First posted 14 Jun 2024cs.CV cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

Transformer architectures rely on position encodings to model the spatial structure of input data. Rotary Position Encoding (RoPE) is a widely used method in language models that encodes relative positions through fixed, block-diagonal, rotation matrices applied to key-query interactions. We hypothesize that this inductive bias limits their RoPE's effectiveness for modalities with high dimensional structure. Lie Relative Encodings (LieRE) introduce a principled generalization of RoPE, aimed at increasing the representational capacity of positional encodings in transformers. Instead of fixed 2D rotations, LieRE learns dense skew-symmetric matrices (Lie algebra elements), which are then differentiable mapped to form high-dimensional rotation matrices (Lie group elements). This results in richer, learnable, and continuous, encodings of both relative and absolute positional information. We demonstrate the effectiveness of LieRE on 2D and 3D vision tasks, showing that it generalizes well to higher input resolutions while maintaining computational efficiency. The code and checkpoints are publicly available at https://github.com/StanfordMIMI/LieRE.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 31 of its 47 references that are in the map. First posted on arXiv 14 Jun 2024.

Builds on

31 of 47 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  8. 2024Mixtral of Expertscited by 1,785
  9. 2021ViViT: A Video Vision Transformercited by 1,610
  10. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  11. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  12. 2018Self-Attention with Relative Position Representationscited by 1,112
19 more
  1. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  2. 2024VMamba: Visual State Space Modelcited by 955
  3. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  4. 2020SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networkscited by 602
  5. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  6. 2022DeiT III: Revenge of the ViTcited by 417
  7. 2019Axial Attention in Multidimensional Transformerscited by 350
  8. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  9. 2016Harmonic Networks: Deep Translation and Rotation Equivariancecited by 293
  10. 2022Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphscited by 258
  11. 2017Learning SO(3) Equivariant Representations with Spherical CNNscited by 244
  12. 2024Rotary Position Embedding for Vision Transformercited by 226
  13. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  14. 2017Polar Transformer Networkscited by 81
  15. 2024Transformers Can Do Arithmetic with the Right Embeddingscited by 77
  16. 2024Contextual Position Encoding: Learning to Count What's Importantcited by 53
  17. 2019Equivariant Transformer Networkscited by 35
  18. 2023Deep Learning Symmetries and Their Lie Groups, Algebras, and Subalgebras from First Principlescited by 22
  19. 2023Lie Group Algebra Convolutional Filterscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 13.

16 papers.

Sort:

By area:

  1. 2025Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AIcited by 8
  2. 2025Group Representational Position Encodingcited by 5
  3. 2025GeoPE:A Unified Geometric Positional Embedding for Structured Tensorscited by 2
  4. 2026Adaptive 3D-RoPE: Physics-Aligned Rotary Positional Encoding for Wireless Foundation Modelscited by 2
  5. 2026Jordan-RoPE: Non-Semisimple Relative Positional Encoding via Complex Jordan Blockscited by 1
  6. 2026LeRoPE: Learnable RoPE Frequencies Improve Language Modelingcited by 1
  7. 2026Parabolic Position Encoding: Vision-Centric, Principled, Extrapolatable, Generalcited by 0
  8. 2026OPRO: Orthogonal Panel-Relative Operators for Panel-Aware In-Context Image Generationcited by 0
  9. 2026Three-Phase Transformercited by 0
  10. 2026The Transformer as a Polar State Estimatorcited by 0
  11. 2026Do Vision Transformers Need All-to-All Attention? Global Communication Through Elastic Learned Corescited by 0
  12. 2026Weierstrass Positional Encoding for Vision Transformerscited by 0
4 more
  1. 2026PJ-RoPE: A Fourier-Jet-Affine Position Space for Relative Attentioncited by 0
  2. 2026The Token Is a Group Element: On Lie-Algebra Attention over Matrix Lie Groupscited by 0
  3. 2026GSF-chi: Global Stereochemical Fields for Chiral Graph Transformerscited by 0
  4. 2026Paragraph Boundaries Are Not White Space:Compression Depth as the Signature of Hierarchical Structurecited by 0