# Hierarchical Mamba Meets Hyperbolic Geometry: A New Paradigm for Structured Language Embeddings

arXiv: https://arxiv.org/abs/2505.18973 · HTML: https://mlsanity.com/paper/2505.18973/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 29 of its 47 references that are in the map.
- First posted on arXiv 25 May 2025.

## Placed in

- State-space sequence models (S4, Mamba) / [Core SSM architectures for language](https://mlsanity.com/ssm/core-ssm-architectures-for-language/)

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [Hyperbolic Graph Convolutional Neural Networks](https://mlsanity.com/paper/1910.12933/) (2019)
- [Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry](https://mlsanity.com/paper/1806.03417/) (2018)
- [Hyperbolic Graph Neural Networks](https://mlsanity.com/paper/1910.12892/) (2019)
- [Hyperbolic Entailment Cones for Learning Hierarchical Embeddings](https://mlsanity.com/paper/1804.01882/) (2018)
- [Poincar'e GloVe: Hyperbolic Word Embeddings](https://mlsanity.com/paper/1810.06546/) (2018)
- [Hyperbolic Deep Neural Networks: A Survey](https://mlsanity.com/paper/2101.04562/) (2021)
- [Geoopt: Riemannian Optimization in PyTorch](https://mlsanity.com/paper/2005.02819/) (2020)
- [Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference](https://mlsanity.com/paper/2403.14520/) (2024)
- [Embedding Text in Hyperbolic Spaces](https://mlsanity.com/paper/1806.04313/) (2018)
- [Probing BERT in Hyperbolic Spaces](https://mlsanity.com/paper/2104.03869/) (2021)
- [The Numerical Stability of Hyperbolic Representation Learning](https://mlsanity.com/paper/2211.00181/) (2022)
- [Language Models as Hierarchy Encoders](https://mlsanity.com/paper/2401.11374/) (2024)
- [Hyperbolic Fine-Tuning for Large Language Models](https://mlsanity.com/paper/2410.04010/) (2024)
- [A Fully Hyperbolic Neural Model for Hierarchical Multi-Class Classification](https://mlsanity.com/paper/2010.02053/) (2020)
- [DeepOnto: A Python Package for Ontology Engineering with Deep Learning](https://mlsanity.com/paper/2307.03067/) (2023)
- [Hyperbolic Learning with Multimodal Large Language Models](https://mlsanity.com/paper/2408.05097/) (2024)
- [SHMamba: Structured Hyperbolic State Space Model for Audio-Visual Question Answering](https://mlsanity.com/paper/2406.09833/) (2024)
- [L^2GC:Lorentzian Linear Graph Convolutional Networks for Node Classification](https://mlsanity.com/paper/2403.06064/) (2024)
- [Hyperbolic sentence representations for solving Textual Entailment](https://mlsanity.com/paper/2406.15472/) (2024)

## Built on by (most cited first, 6 of 6)

- [Hyperbolic Large Language Models](https://mlsanity.com/paper/2509.05757/) (2025)
- [Hierarchical Latent Structures in Data Generation Process Unify Mechanistic Phenomena across Scale](https://mlsanity.com/paper/2603.06592/) (2026)
- [GTR-Mamba: Geometry-to-Tangent Routing Mamba for Hyperbolic POI Recommendation](https://mlsanity.com/paper/2510.22942/) (2025)
- [HypEHR: Hyperbolic Modeling of Electronic Health Records for Efficient Question Answering](https://mlsanity.com/paper/2604.21027/) (2026)
- [Hesitation Has a Geometry: Entropy-Trained Hyperbolic Probes for Sparse Activation Steering](https://mlsanity.com/paper/2610.02391/) (2026)
- [Hyperbolic Time Series Forecasting with Learnable Decomposition and Manifold Dynamics](https://mlsanity.com/paper/2610.15760/) (2026)

## Abstract (arXiv)

Selective state-space models excel at long-sequence modeling, but their capacity for language representation -- in complex hierarchical reasoning -- remains underexplored. Most large language models rely on textitflat Euclidean embeddings, limiting their ability to capture latent hierarchies. To address this, we propose it Hierarchical Mamba (HiM), integrating efficient Mamba2 with hyperbolic geometry to learn hierarchy-aware language embeddings for deeper linguistic understanding. Mamba2-processed sequences are projected to the Poincar'e ball or Lorentzian manifold with ``learnable'' curvature, optimized with a hyperbolic loss. Our HiM model facilitates the capture of relational distances across varying hierarchical levels, enabling effective long-range reasoning for tasks like mixed-hop prediction and multi-hop inference in hierarchical classification. Experimental results show both HiM variants effectively capture hierarchical relationships across four linguistic and medical datasets, surpassing Euclidean baselines, with HiM-Poincar'e providing fine-grained distinctions with higher h-norms, while HiM-Lorentz offers more stable, compact, and hierarchy-preserving embeddings-favoring robustness. The source code is publicly available at https://github.com/BerryByte/HiM.