# GridPE: A Grid Cell-Inspired Unified Position Embedding for Arbitrary-Dimensional Spaces

arXiv: https://arxiv.org/abs/2406.07049 · HTML: https://mlsanity.com/paper/2406.07049/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 50 references that are in the map.
- First posted on arXiv 11 Jun 2024.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [PCT: Point cloud transformer](https://mlsanity.com/paper/2012.09688/) (2020)
- [EVA-02: A Visual Representation for Neon Genesis](https://mlsanity.com/paper/2303.11331/) (2023)
- [Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action](https://mlsanity.com/paper/2312.17172/) (2023)
- [Rotary Position Embedding for Vision Transformer](https://mlsanity.com/paper/2403.13298/) (2024)
- [Learnable Fourier Features for Multi-Dimensional Spatial Positional Encoding](https://mlsanity.com/paper/2106.02795/) (2021)
- [Multi-Scale Representation Learning for Spatial Feature Distributions using Grid Cells](https://mlsanity.com/paper/2003.00824/) (2020)
- [A Review of Location Encoding for GeoAI: Methods and Applications](https://mlsanity.com/paper/2111.04006/) (2021)
- [Encoding word order in complex embeddings](https://mlsanity.com/paper/1912.12333/) (2019)
- [Sphere2Vec: A General-Purpose Location Representation Learning over a Spherical Surface for Large-Scale Geospatial Predictions](https://mlsanity.com/paper/2306.17624/) (2023)
- [Rethinking Positional Encoding](https://mlsanity.com/paper/2107.02561/) (2021)
- [SHAPE: Shifted Absolute Position Embedding for Transformers](https://mlsanity.com/paper/2109.05644/) (2021)
- [Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encoding](https://mlsanity.com/paper/2312.17044/) (2023)
- [Persformer: A Transformer Architecture for Topological Machine Learning](https://mlsanity.com/paper/2112.15210/) (2021)
- [Learning Grid Cells as Vector Representation of Self-Position Coupled with Matrix Representation of Self-Motion](https://mlsanity.com/paper/1810.05597/) (2018)
- [The Sense of Place: Grid Cells in the Brain and the Transcendental Number e](https://mlsanity.com/paper/1304.0031/) (2013)
- [A computational model for grid maps in neural populations](https://mlsanity.com/paper/1902.06553/) (2019)

## Abstract (arXiv)

Understanding spatial relationships across all dimensions is fundamental for intelligent systems. However, existing positional embeddings, such as Rotary Positional Embedding (RoPE), lack theoretical guarantees for high-dimensional spatiotemporal tasks like video understanding and robotic navigation. Inspired by the hexagonal periodic coding of grid cells in mammalian spatial cognition, we propose GridPE -- a novel positional embedding framework that integrates computational neuroscience principles with harmonic analysis. Our approach builds upon Random Fourier Features and leverages principles from neuroscience to construct efficient embeddings. Theoretically, we prove that any translation-invariant spatial function can be approximated by a finite sum of Fourier bases, which naturally reduces to RoPE in the one-dimensional case. We then derive the directions and quantities of frequency vectors at each scale in any Euclidean dimension, along with the optimal ratio between different scales, from a bioavailability perspective. These derivations are equivalent to the relationship between the centroid and the vertices of a regular simplex in that dimension. We validate GridPE across a range of spatial modeling tasks, including 2D image classification (ImageNet100) and 3D point cloud recognition (ModelNet40). Our theoretical analysis establishes GridPE as a unified framework for positional embedding in arbitrary-dimensional spaces, while empirical results demonstrate its superiority over existing methods.