# Hyperbolic Attention Networks

arXiv: https://arxiv.org/abs/1805.09786 · HTML: https://mlsanity.com/paper/1805.09786/

## Key facts

- Cited by 122 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 May 2018.

## Built on by (most cited first, 30 of 122)

- [Hyperbolic Deep Neural Networks: A Survey](https://mlsanity.com/paper/2101.04562/) (2021)
- [Fully Hyperbolic Neural Networks](https://mlsanity.com/paper/2105.14686/) (2021)
- [Hyperbolic Deep Learning in Computer Vision: A Survey](https://mlsanity.com/paper/2305.06611/) (2023)
- [Lorentzian Graph Convolutional Networks](https://mlsanity.com/paper/2104.07477/) (2021)
- [Discrete-time Temporal Network Embedding via Implicit Hierarchical Learning in Hyperbolic Space](https://mlsanity.com/paper/2107.03767/) (2021)
- [Clipped Hyperbolic Classifiers Are Super-Hyperbolic Classifiers](https://mlsanity.com/paper/2107.11472/) (2021)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [Hyperbolic Variational Graph Neural Network for Modeling Dynamic Graphs](https://mlsanity.com/paper/2104.02228/) (2021)
- [Learning the Predictability of the Future](https://mlsanity.com/paper/2101.01600/) (2021)
- [HYperbolic Self-Paced Learning for Self-Supervised Skeleton-based Action Representations](https://mlsanity.com/paper/2303.06242/) (2023)
- [Hyperbolic Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/2202.13852/) (2022)
- [A Self-supervised Mixed-curvature Graph Neural Network](https://mlsanity.com/paper/2112.05393/) (2021)
- [Self-Supervised Continual Graph Learning in Adaptive Riemannian Spaces](https://mlsanity.com/paper/2211.17068/) (2022)
- [CO-SNE: Dimensionality Reduction and Visualization for Hyperbolic Data](https://mlsanity.com/paper/2111.15037/) (2021)
- [Hyperbolic Fine-Tuning for Large Language Models](https://mlsanity.com/paper/2410.04010/) (2024)
- [Learning Conjoint Attentions for Graph Neural Nets](https://mlsanity.com/paper/2102.03147/) (2021)
- [HELM: Hyperbolic Large Language Models via Mixture-of-Curvature Experts](https://mlsanity.com/paper/2505.24722/) (2025)
- [A Self-supervised Riemannian GNN with Time Varying Curvature for Temporal Graph Learning](https://mlsanity.com/paper/2208.14073/) (2022)
- [Heterogeneous manifolds for curvature-aware graph embedding](https://mlsanity.com/paper/2202.01185/) (2022)
- [Hyperbolic Representation Learning: Revisiting and Advancing](https://mlsanity.com/paper/2306.09118/) (2023)
- [Riemannian Residual Neural Networks](https://mlsanity.com/paper/2310.10013/) (2023)
- [Beyond Euclid: An Illustrated Guide to Modern Machine Learning with Geometric, Topological, and Algebraic Structures](https://mlsanity.com/paper/2407.09468/) (2024)
- [Hyperbolic Category Discovery](https://mlsanity.com/paper/2504.06120/) (2025)
- [Hyperbolic Deep Learning for Foundation Models: A Survey](https://mlsanity.com/paper/2507.17787/) (2025)
- [Multi-modal Entity Alignment in Hyperbolic Space](https://mlsanity.com/paper/2106.03619/) (2021)
- [Hyperbolic Audio-visual Zero-shot Learning](https://mlsanity.com/paper/2308.12558/) (2023)
- [Hyperbolic Active Learning for Semantic Segmentation under Domain Shift](https://mlsanity.com/paper/2306.11180/) (2023)
- [Position: Beyond Euclidean -- Foundation Models Should Embrace Non-Euclidean Geometries](https://mlsanity.com/paper/2504.08896/) (2025)
- [Learning Structured Representations with Hyperbolic Embeddings](https://mlsanity.com/paper/2412.01023/) (2024)
- [Unsupervised Hyperbolic Representation Learning via Message Passing Auto-Encoders](https://mlsanity.com/paper/2103.16046/) (2021)

## Abstract (arXiv)

We introduce hyperbolic attention networks to endow neural networks with enough capacity to match the complexity of data with hierarchical and power-law structure. A few recent approaches have successfully demonstrated the benefits of imposing hyperbolic geometry on the parameters of shallow networks. We extend this line of work by imposing hyperbolic geometry on the activations of neural networks. This allows us to exploit hyperbolic geometry to reason about embeddings produced by deep networks. We achieve this by re-expressing the ubiquitous mechanism of soft attention in terms of operations defined for hyperboloid and Klein models. Our method shows improvements in terms of generalization on neural machine translation, learning on graphs and visual question answering tasks while keeping the neural representations compact.