# LieRE: Lie Rotational Positional Encodings

arXiv: https://arxiv.org/abs/2406.10322 · HTML: https://mlsanity.com/paper/2406.10322/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 47 references that are in the map.
- First posted on arXiv 14 Jun 2024.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [VMamba: Visual State Space Model](https://mlsanity.com/paper/2401.10166/) (2024)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks](https://mlsanity.com/paper/2006.10503/) (2020)
- [YaRN: Efficient Context Window Extension of Large Language Models](https://mlsanity.com/paper/2309.00071/) (2023)
- [DeiT III: Revenge of the ViT](https://mlsanity.com/paper/2204.07118/) (2022)
- [Axial Attention in Multidimensional Transformers](https://mlsanity.com/paper/1912.12180/) (2019)
- [LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens](https://mlsanity.com/paper/2402.13753/) (2024)
- [Harmonic Networks: Deep Translation and Rotation Equivariance](https://mlsanity.com/paper/1612.04642/) (2016)
- [Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs](https://mlsanity.com/paper/2206.11990/) (2022)
- [Learning SO(3) Equivariant Representations with Spherical CNNs](https://mlsanity.com/paper/1711.06721/) (2017)
- [Rotary Position Embedding for Vision Transformer](https://mlsanity.com/paper/2403.13298/) (2024)
- [Focused Transformer: Contrastive Training for Context Scaling](https://mlsanity.com/paper/2307.03170/) (2023)
- [Polar Transformer Networks](https://mlsanity.com/paper/1709.01889/) (2017)
- [Transformers Can Do Arithmetic with the Right Embeddings](https://mlsanity.com/paper/2405.17399/) (2024)
- [Contextual Position Encoding: Learning to Count What's Important](https://mlsanity.com/paper/2405.18719/) (2024)
- [Equivariant Transformer Networks](https://mlsanity.com/paper/1901.11399/) (2019)
- [Deep Learning Symmetries and Their Lie Groups, Algebras, and Subalgebras from First Principles](https://mlsanity.com/paper/2301.05638/) (2023)

## Built on by (most cited first, 18 of 18)

- [Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AI](https://mlsanity.com/paper/2510.16720/) (2025)
- [Group Representational Position Encoding](https://mlsanity.com/paper/2512.07805/) (2025)
- [GeoPE:A Unified Geometric Positional Embedding for Structured Tensors](https://mlsanity.com/paper/2512.04963/) (2025)
- [Adaptive 3D-RoPE: Physics-Aligned Rotary Positional Encoding for Wireless Foundation Models](https://mlsanity.com/paper/2605.00968/) (2026)
- [Jordan-RoPE: Non-Semisimple Relative Positional Encoding via Complex Jordan Blocks](https://mlsanity.com/paper/2605.04217/) (2026)
- [LeRoPE: Learnable RoPE Frequencies Improve Language Modeling](https://mlsanity.com/paper/2607.10134/) (2026)
- [Power law graph attention: exact generalization of scaled dot-product attention, empirical collapse at inference](https://mlsanity.com/paper/2608.10288/) (2026)
- [Parabolic Position Encoding: Vision-Centric, Principled, Extrapolatable, General](https://mlsanity.com/paper/2602.01418/) (2026)
- [OPRO: Orthogonal Panel-Relative Operators for Panel-Aware In-Context Image Generation](https://mlsanity.com/paper/2603.27637/) (2026)
- [Three-Phase Transformer](https://mlsanity.com/paper/2604.14430/) (2026)
- [The Transformer as a Polar State Estimator](https://mlsanity.com/paper/2605.11007/) (2026)
- [Do Vision Transformers Need All-to-All Attention? Global Communication Through Elastic Learned Cores](https://mlsanity.com/paper/2605.12491/) (2026)
- [Weierstrass Positional Encoding for Vision Transformers](https://mlsanity.com/paper/2605.23719/) (2026)
- [PJ-RoPE: A Fourier-Jet-Affine Position Space for Relative Attention](https://mlsanity.com/paper/2606.05345/) (2026)
- [The Token Is a Group Element: On Lie-Algebra Attention over Matrix Lie Groups](https://mlsanity.com/paper/2606.20547/) (2026)
- [GSF-chi: Global Stereochemical Fields for Chiral Graph Transformers](https://mlsanity.com/paper/2609.12532/) (2026)
- [Paragraph Boundaries Are Not White Space:Compression Depth as the Signature of Hierarchical Structure](https://mlsanity.com/paper/2609.23551/) (2026)
- [WorldRoPE: Probabilistic World-Centric Rotary Embedding for Multi-View Attention](https://mlsanity.com/paper/2610.16337/) (2026)

## Abstract (arXiv)

Transformer architectures rely on position encodings to model the spatial structure of input data. Rotary Position Encoding (RoPE) is a widely used method in language models that encodes relative positions through fixed, block-diagonal, rotation matrices applied to key-query interactions. We hypothesize that this inductive bias limits their RoPE's effectiveness for modalities with high dimensional structure. Lie Relative Encodings (LieRE) introduce a principled generalization of RoPE, aimed at increasing the representational capacity of positional encodings in transformers. Instead of fixed 2D rotations, LieRE learns dense skew-symmetric matrices (Lie algebra elements), which are then differentiable mapped to form high-dimensional rotation matrices (Lie group elements). This results in richer, learnable, and continuous, encodings of both relative and absolute positional information. We demonstrate the effectiveness of LieRE on 2D and 3D vision tasks, showing that it generalizes well to higher input resolutions while maintaining computational efficiency. The code and checkpoints are publicly available at https://github.com/StanfordMIMI/LieRE.