# Hypformer: Exploring Efficient Transformer Fully in Hyperbolic Space

arXiv: https://arxiv.org/abs/2407.01290 · HTML: https://mlsanity.com/paper/2407.01290/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 92 references that are in the map.
- First posted on arXiv 1 Jul 2024.

## Builds on (most cited first, 30 of 56)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Simplifying Graph Convolutional Networks](https://mlsanity.com/paper/1902.07153/) (2019)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [Recipe for a General, Powerful, Scalable Graph Transformer](https://mlsanity.com/paper/2205.12454/) (2022)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [Hypergraph Neural Networks](https://mlsanity.com/paper/1809.09401/) (2018)
- [Efficient Content-Based Sparse Attention with Routing Transformers](https://mlsanity.com/paper/2003.05997/) (2020)
- [Hyperbolic Neural Networks](https://mlsanity.com/paper/1805.09112/) (2018)
- [Hyperbolic Graph Convolutional Neural Networks](https://mlsanity.com/paper/1910.12933/) (2019)
- [SIGN: Scalable Inception Graph Neural Networks](https://mlsanity.com/paper/2004.11198/) (2020)
- [Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry](https://mlsanity.com/paper/1806.03417/) (2018)
- [Hyperbolic Graph Neural Networks](https://mlsanity.com/paper/1910.12892/) (2019)
- [Learning Discrete Structures for Graph Neural Networks](https://mlsanity.com/paper/1903.11960/) (2019)
- [Hyperbolic Image Embeddings](https://mlsanity.com/paper/1904.02239/) (2019)
- [Iterative Deep Graph Learning for Graph Neural Networks: Better and Robust Node Embeddings](https://mlsanity.com/paper/2006.13009/) (2020)
- [NodeFormer: A Scalable Graph Structure Learning Transformer for Node Classification](https://mlsanity.com/paper/2306.08385/) (2023)
- [Low-Dimensional Hyperbolic Knowledge Graph Embeddings](https://mlsanity.com/paper/2005.00545/) (2020)
- [Representing Long-Range Context for Graph Neural Networks with Global Attention](https://mlsanity.com/paper/2201.08821/) (2022)
- [Pure Transformers are Powerful Graph Learners](https://mlsanity.com/paper/2207.02505/) (2022)
- [FLatten Transformer: Vision Transformer using Focused Linear Attention](https://mlsanity.com/paper/2308.00442/) (2023)
- [Hyperbolic Entailment Cones for Learning Hierarchical Embeddings](https://mlsanity.com/paper/1804.01882/) (2018)

## Built on by (most cited first, 30 of 43)

- [RiemannGFM: Learning a Graph Foundation Model from Riemannian Geometry](https://mlsanity.com/paper/2502.03251/) (2025)
- [Hyperbolic Fine-Tuning for Large Language Models](https://mlsanity.com/paper/2410.04010/) (2024)
- [HELM: Hyperbolic Large Language Models via Mixture-of-Curvature Experts](https://mlsanity.com/paper/2505.24722/) (2025)
- [Hyperbolic Deep Learning for Foundation Models: A Survey](https://mlsanity.com/paper/2507.17787/) (2025)
- [Position: Beyond Euclidean -- Foundation Models Should Embrace Non-Euclidean Geometries](https://mlsanity.com/paper/2504.08896/) (2025)
- [Lorentzian Residual Neural Networks](https://mlsanity.com/paper/2412.14695/) (2024)
- [HyperCore: The Core Framework for Building Hyperbolic Foundation Models with Comprehensive Modules](https://mlsanity.com/paper/2504.08912/) (2025)
- [Geo-Sign: Hyperbolic Contrastive Regularisation for Geometrically Aware Sign Language Translation](https://mlsanity.com/paper/2506.00129/) (2025)
- [Learning to Normalize on the SPD Manifold under Bures-Wasserstein Geometry](https://mlsanity.com/paper/2504.00660/) (2025)
- [Hyperbolic Large Language Models](https://mlsanity.com/paper/2509.05757/) (2025)
- [Klein Model for Hyperbolic Neural Networks](https://mlsanity.com/paper/2410.16813/) (2024)
- [Hyperbolic Busemann Neural Networks](https://mlsanity.com/paper/2602.18858/) (2026)
- [HyLiFormer: Hyperbolic Linear Attention for Skeleton-based Human Action Recognition](https://mlsanity.com/paper/2502.05869/) (2025)
- [GGBall: Graph Generative Model on Poincar'e Ball](https://mlsanity.com/paper/2506.07198/) (2025)
- [Fast and Geometrically Grounded Lorentz Neural Networks](https://mlsanity.com/paper/2601.21529/) (2026)
- [Intrinsic Lorentz Neural Network](https://mlsanity.com/paper/2602.23981/) (2026)
- [Emotion Collider: Dual Hyperbolic Mirror Manifolds for Sentiment Recovery via Anti Emotion Reflection](https://mlsanity.com/paper/2602.16161/) (2026)
- [EEG-Based Multimodal Learning via Hyperbolic Mixture-of-Curvature Experts](https://mlsanity.com/paper/2604.12579/) (2026)
- [Hyperbolic Multimodal Continual Learning](https://mlsanity.com/paper/2608.09572/) (2026)
- [FlatLand: Personalized Graph Federated Learning via Tailored Lorentz Space](https://mlsanity.com/paper/2608.21096/) (2026)
- [Towards Non-Euclidean Foundation Models: Advancing AI Beyond Euclidean Frameworks](https://mlsanity.com/paper/2505.14417/) (2025)
- [Hyperbolic Cycle Alignment for Infrared-Visible Image Fusion](https://mlsanity.com/paper/2507.23508/) (2025)
- [HyPCV-Former: Hyperbolic Spatio-Temporal Transformer for 3D Point Cloud Video Anomaly Detection](https://mlsanity.com/paper/2508.00473/) (2025)
- [CAT: Curvature-Adaptive Transformers for Geometry-Aware Learning](https://mlsanity.com/paper/2510.01634/) (2025)
- [Randomized HyperSteiner: A Stochastic Delaunay Triangulation Heuristic for the Hyperbolic Steiner Minimal Tree](https://mlsanity.com/paper/2510.09328/) (2025)
- [HexFormer: Hyperbolic Vision Transformer with Exponential Map Aggregation](https://mlsanity.com/paper/2601.19849/) (2026)
- [HypRAG: Hyperbolic Dense Retrieval for Retrieval Augmented Generation](https://mlsanity.com/paper/2602.07739/) (2026)
- [ARGENT: Adaptive Hierarchical Image-Text Representations](https://mlsanity.com/paper/2603.23311/) (2026)
- [Memory Has Geometry: Non-Uniform Geometric Memory for Long-Horizon Personalized AI](https://mlsanity.com/paper/2609.17969/) (2026)
- [hyperbolix: Hyperbolic Deep Learning in JAX](https://mlsanity.com/paper/2609.28248/) (2026)

## Abstract (arXiv)

Hyperbolic geometry have shown significant potential in modeling complex structured data, particularly those with underlying tree-like and hierarchical structures. Despite the impressive performance of various hyperbolic neural networks across numerous domains, research on adapting the Transformer to hyperbolic space remains limited. Previous attempts have mainly focused on modifying self-attention modules in the Transformer. However, these efforts have fallen short of developing a complete hyperbolic Transformer. This stems primarily from: (i) the absence of well-defined modules in hyperbolic space, including linear transformation layers, LayerNorm layers, activation functions, dropout operations, etc. (ii) the quadratic time complexity of the existing hyperbolic self-attention module w.r.t the number of input tokens, which hinders its scalability. To address these challenges, we propose, Hypformer, a novel hyperbolic Transformer based on the Lorentz model of hyperbolic geometry. In Hypformer, we introduce two foundational blocks that define the essential modules of the Transformer in hyperbolic space. Furthermore, we develop a linear self-attention mechanism in hyperbolic space, enabling hyperbolic Transformer to process billion-scale graph data and long-sequence inputs for the first time. Our experimental results confirm the effectiveness and efficiency of Hypformer across various datasets, demonstrating its potential as an effective and scalable solution for large-scale data representation and large models.