# RiemannFormer: A Framework for Attention in Curved Spaces

arXiv: https://arxiv.org/abs/2506.07405 · HTML: https://mlsanity.com/paper/2506.07405/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 16 references that are in the map.
- First posted on arXiv 9 Jun 2025.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [Conditional Positional Encodings for Vision Transformers](https://mlsanity.com/paper/2102.10882/) (2021)
- [Rotary Position Embedding for Vision Transformer](https://mlsanity.com/paper/2403.13298/) (2024)
- [Rethinking Positional Encoding in Language Pre-training](https://mlsanity.com/paper/2006.15595/) (2020)
- [A Length-Extrapolatable Transformer](https://mlsanity.com/paper/2212.10554/) (2022)
- [Contextual Position Encoding: Learning to Count What's Important](https://mlsanity.com/paper/2405.18719/) (2024)
- [DAPE: Data-Adaptive Positional Encoding for Length Extrapolation](https://mlsanity.com/paper/2405.14722/) (2024)

## Built on by (most cited first, 8 of 8)

- [CliffordNet: All You Need is Geometric Algebra](https://mlsanity.com/paper/2601.06793/) (2026)
- [The Curved Spacetime of Transformer Architectures](https://mlsanity.com/paper/2511.03060/) (2025)
- [IsoQuant: Hardware-Aligned SO(4) Isoclinic Rotations for LLM KV Cache Compression](https://mlsanity.com/paper/2603.28430/) (2026)
- [The Transformer as a Polar State Estimator](https://mlsanity.com/paper/2605.11007/) (2026)
- [The Token Is a Group Element: On Lie-Algebra Attention over Matrix Lie Groups](https://mlsanity.com/paper/2606.20547/) (2026)
- [Riemannian Geometry for Pre-trained Language Model Embeddings](https://mlsanity.com/paper/2607.07047/) (2026)
- [Riemannian Attention Mechanisms for Transformers: A Theoretical Framework and Architecture Design](https://mlsanity.com/paper/2608.01283/) (2026)
- [QSV: Quat-Sphere-Vision for Coupled Quaternion Attention on Spherical Lattices](https://mlsanity.com/paper/2609.30592/) (2026)

## Abstract (arXiv)

This research endeavors to offer insights into unlocking the further potential of transformer-based architectures. One of the primary motivations is to offer a geometric interpretation for the attention mechanism in transformers. In our framework, the attention mainly involves metric tensors, tangent spaces, inner product, and how they relate to each other. These quantities and structures at discrete positions are intricately interconnected via the parallel transport of tangent vectors. To make the learning process more efficient, we reduce the number of parameters through ingenious predefined configurations. Moreover, we introduce an explicit mechanism to highlight a neighborhood by attenuating the remote values, given that transformers inherently neglect local inductive bias. Experimental results demonstrate that our modules deliver significant performance improvements relative to the baseline. More evaluation experiments on visual and large language models will be launched successively.