# HELM: Hyperbolic Large Language Models via Mixture-of-Curvature Experts

arXiv: https://arxiv.org/abs/2505.24722 · HTML: https://mlsanity.com/paper/2505.24722/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 49 references that are in the map.
- First posted on arXiv 30 May 2025.

## Builds on (most cited first, 30 of 41)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2401.06066/) (2024)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [Hyperbolic Neural Networks](https://mlsanity.com/paper/1805.09112/) (2018)
- [Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry](https://mlsanity.com/paper/1806.03417/) (2018)
- [Hyperbolic Entailment Cones for Learning Hierarchical Embeddings](https://mlsanity.com/paper/1804.01882/) (2018)
- [Poincar'e GloVe: Hyperbolic Word Embeddings](https://mlsanity.com/paper/1810.06546/) (2018)
- [Hyperbolic Image-Text Representations](https://mlsanity.com/paper/2304.09172/) (2023)
- [Hyperbolic Attention Networks](https://mlsanity.com/paper/1805.09786/) (2018)
- [Round and Round We Go! What makes Rotary Positional Encodings useful?](https://mlsanity.com/paper/2410.06205/) (2024)
- [Fully Hyperbolic Neural Networks](https://mlsanity.com/paper/2105.14686/) (2021)
- [Compositional Entailment Learning for Hyperbolic Vision-Language Models](https://mlsanity.com/paper/2410.06912/) (2024)
- [Hyperbolic Contrastive Learning for Visual Representations beyond Objects](https://mlsanity.com/paper/2212.00653/) (2022)
- [Fully Hyperbolic Convolutional Neural Networks for Computer Vision](https://mlsanity.com/paper/2303.15919/) (2023)

## Built on by (most cited first, 26 of 26)

- [Hyperbolic Large Language Models](https://mlsanity.com/paper/2509.05757/) (2025)
- [Hyperbolic Busemann Neural Networks](https://mlsanity.com/paper/2602.18858/) (2026)
- [Modality Alignment across Trees on Heterogeneous Hyperbolic Manifolds](https://mlsanity.com/paper/2510.27391/) (2025)
- [Fast and Geometrically Grounded Lorentz Neural Networks](https://mlsanity.com/paper/2601.21529/) (2026)
- [Intrinsic Lorentz Neural Network](https://mlsanity.com/paper/2602.23981/) (2026)
- [HyperbolicRAG: Enhancing Retrieval-Augmented Generation with Hyperbolic Representations](https://mlsanity.com/paper/2511.18808/) (2025)
- [EEG-Based Multimodal Learning via Hyperbolic Mixture-of-Curvature Experts](https://mlsanity.com/paper/2604.12579/) (2026)
- [Randomized HyperSteiner: A Stochastic Delaunay Triangulation Heuristic for the Hyperbolic Steiner Minimal Tree](https://mlsanity.com/paper/2510.09328/) (2025)
- [HypRAG: Hyperbolic Dense Retrieval for Retrieval Augmented Generation](https://mlsanity.com/paper/2602.07739/) (2026)
- [Parameter-Efficient Fine-Tuning of LLMs with Mixture of Space Experts](https://mlsanity.com/paper/2602.14490/) (2026)
- [HypEHR: Hyperbolic Modeling of Electronic Health Records for Efficient Question Answering](https://mlsanity.com/paper/2604.21027/) (2026)
- [hyperbolix: Hyperbolic Deep Learning in JAX](https://mlsanity.com/paper/2609.28248/) (2026)
- [HyperHELM: Hyperbolic Hierarchy Encoding for mRNA Language Modeling](https://mlsanity.com/paper/2509.24655/) (2025)
- [Combining Euclidean and Hyperbolic Representations for Node-level Anomaly Detection](https://mlsanity.com/paper/2510.11827/) (2025)
- [HyFL-CLIP: Hyperbolic Fine-Tuning of CLIP for Robust Long-Context Understanding](https://mlsanity.com/paper/2607.00428/) (2026)
- [Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Models](https://mlsanity.com/paper/2607.02909/) (2026)
- [After the Euclidean Highway: Hyperbolic Expert AI as the Next Innovation](https://mlsanity.com/paper/2607.17513/) (2026)
- [Riemannian Attention Mechanisms for Transformers: A Theoretical Framework and Architecture Design](https://mlsanity.com/paper/2608.01283/) (2026)
- [Real-Time Hybrid Retrieval in Hyperbolic Space for Retrieval-Augmented Generation on Edge Devices](https://mlsanity.com/paper/2608.01450/) (2026)
- [A Hyperbolicity Atlas of Large Language Model Hidden States](https://mlsanity.com/paper/2609.07053/) (2026)
- [Geometry-Aware Hyperbolic Residual-Quantized Variational Autoencoders](https://mlsanity.com/paper/2609.26342/) (2026)
- [Rethinking the Fully Hyperbolic Vision Transformer in Polar Coordinates](https://mlsanity.com/paper/2609.32899/) (2026)
- [Explaining Hyperbolic Neural Networks via Geometry-Aware Relevance Propagation](https://mlsanity.com/paper/2609.35128/) (2026)
- [Hesitation Has a Geometry: Entropy-Trained Hyperbolic Probes for Sparse Activation Steering](https://mlsanity.com/paper/2610.02391/) (2026)
- [Constant-Curvature Sliced Gromov-Wasserstein for Heterogeneous Cross-Curvature Alignment](https://mlsanity.com/paper/2610.07218/) (2026)
- [Hyperbolic Time Series Forecasting with Learnable Decomposition and Manifold Dynamics](https://mlsanity.com/paper/2610.15760/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have shown great success in text modeling tasks across domains. However, natural language exhibits inherent semantic hierarchies and nuanced geometric structure, which current LLMs do not capture completely owing to their reliance on Euclidean operations. Recent studies have also shown that not respecting the geometry of token embeddings leads to training instabilities and degradation of generative capabilities. These findings suggest that shifting to non-Euclidean geometries can better align language models with the underlying geometry of text. We thus propose to operate fully in Hyperbolic space, known for its expansive, scale-free, and low-distortion properties. We thus introduce HELM, a family of HypErbolic Large Language Models, offering a geometric rethinking of the Transformer-based LLM that addresses the representational inflexibility, missing set of necessary operations, and poor scalability of existing hyperbolic LMs. We additionally introduce a Mixture-of-Curvature Experts model, HELM-MICE, where each expert operates in a distinct curvature space to encode more fine-grained geometric structure from text, as well as a dense model, HELM-D. For HELM-MICE, we further develop hyperbolic Multi-Head Latent Attention (HMLA) for efficient, reduced-KV-cache training and inference. For both models, we develop essential hyperbolic equivalents of rotary positional encodings and RMS normalization. We are the first to train fully hyperbolic LLMs at billion-parameter scale, and evaluate them on well-known benchmarks such as MMLU and ARC, spanning STEM problem-solving, general knowledge, and commonsense reasoning. Our results show consistent gains from our HELM architectures -- up to 4% -- over popular Euclidean architectures used in LLaMA and DeepSeek, highlighting the efficacy and enhanced reasoning afforded by hyperbolic geometry in large-scale LM pretraining.