# Hyperbolic Deep Learning for Foundation Models: A Survey

arXiv: https://arxiv.org/abs/2507.17787 · HTML: https://mlsanity.com/paper/2507.17787/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 66 of its 110 references that are in the map.
- First posted on arXiv 23 Jul 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 30 of 66)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://mlsanity.com/paper/2404.16130/) (2024)
- [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [Simple and Deep Graph Convolutional Networks](https://mlsanity.com/paper/2007.02133/) (2020)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [An Introduction to Convolutional Neural Networks](https://mlsanity.com/paper/1511.08458/) (2015)
- [Dense Contrastive Learning for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2011.09157/) (2020)
- [Hyperbolic Neural Networks](https://mlsanity.com/paper/1805.09112/) (2018)
- [Hyperbolic Graph Convolutional Neural Networks](https://mlsanity.com/paper/1910.12933/) (2019)
- [Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry](https://mlsanity.com/paper/1806.03417/) (2018)
- [Hyperbolic Graph Neural Networks](https://mlsanity.com/paper/1910.12892/) (2019)
- [Visualizing and Measuring the Geometry of BERT](https://mlsanity.com/paper/1906.02715/) (2019)
- [Representation Tradeoffs for Hyperbolic Embeddings](https://mlsanity.com/paper/1804.03329/) (2018)
- [Convergence for score-based generative modeling with polynomial complexity](https://mlsanity.com/paper/2206.06227/) (2022)
- [Hyperbolic Entailment Cones for Learning Hierarchical Embeddings](https://mlsanity.com/paper/1804.01882/) (2018)

## Built on by (most cited first, 20 of 20)

- [Fast and Geometrically Grounded Lorentz Neural Networks](https://mlsanity.com/paper/2601.21529/) (2026)
- [Angular Gradient Sign Method: Uncovering Vulnerabilities in Hyperbolic Networks](https://mlsanity.com/paper/2511.12985/) (2025)
- [HyperbolicRAG: Enhancing Retrieval-Augmented Generation with Hyperbolic Representations](https://mlsanity.com/paper/2511.18808/) (2025)
- [Uncertainty-guided Compositional Alignment with Part-to-Whole Semantic Representativeness in Hyperbolic Vision-Language Models](https://mlsanity.com/paper/2603.22042/) (2026)
- [Randomized HyperSteiner: A Stochastic Delaunay Triangulation Heuristic for the Hyperbolic Steiner Minimal Tree](https://mlsanity.com/paper/2510.09328/) (2025)
- [HypRAG: Hyperbolic Dense Retrieval for Retrieval Augmented Generation](https://mlsanity.com/paper/2602.07739/) (2026)
- [A Neural Tension Operator for Curve Subdivision across Constant Curvature Geometries](https://mlsanity.com/paper/2603.28937/) (2026)
- [HypEHR: Hyperbolic Modeling of Electronic Health Records for Efficient Question Answering](https://mlsanity.com/paper/2604.21027/) (2026)
- [The Gate Always Closes: On Injecting Auxiliary Signals into Frozen Vision-Language Models](https://mlsanity.com/paper/2607.23335/) (2026)
- [Riemann GeoResolver: A Non-Euclidean Attention Framework from Euclidean Resolver to Hyperbolic-Spherical Geometry](https://mlsanity.com/paper/2608.10416/) (2026)
- [hyperbolix: Hyperbolic Deep Learning in JAX](https://mlsanity.com/paper/2609.28248/) (2026)
- [More Than A Shortcut: A Hyperbolic Approach To Early-Exit Networks](https://mlsanity.com/paper/2511.00641/) (2025)
- [Tutorial on Reasoning for IR & IR for Reasoning](https://mlsanity.com/paper/2602.03640/) (2026)
- [HYDRA: Hyperbolic Dynamic Representation Architecture for Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2608.12194/) (2026)
- [Adaptive Interpolatory Curve Subdivision with Learned Local Angles](https://mlsanity.com/paper/2609.17566/) (2026)
- [Hierarchical Prompt Learning for Hyperbolic Vision-Language Models](https://mlsanity.com/paper/2609.24276/) (2026)
- [Geometry-Aware Hyperbolic Residual-Quantized Variational Autoencoders](https://mlsanity.com/paper/2609.26342/) (2026)
- [Rethinking the Fully Hyperbolic Vision Transformer in Polar Coordinates](https://mlsanity.com/paper/2609.32899/) (2026)
- [Explaining Hyperbolic Neural Networks via Geometry-Aware Relevance Propagation](https://mlsanity.com/paper/2609.35128/) (2026)
- [CurvSpec: Adaptive Multi-Curvature Learning for Partial Relevant Video Retrieval](https://mlsanity.com/paper/2609.36815/) (2026)

## Abstract (arXiv)

Foundation models pre-trained on massive datasets, including large language models (LLMs), vision-language models (VLMs), and large multimodal models, have demonstrated remarkable success in diverse downstream tasks. However, recent studies have shown fundamental limitations of these models: (1) limited representational capacity, (2) lower adaptability, and (3) diminishing scalability. These shortcomings raise a critical question: is Euclidean geometry truly the optimal inductive bias for all foundation models, or could incorporating alternative geometric spaces enable models to better align with the intrinsic structure of real-world data and improve reasoning processes? Hyperbolic spaces, a class of non-Euclidean manifolds characterized by exponential volume growth with respect to distance, offer a mathematically grounded solution. These spaces enable low-distortion embeddings of hierarchical structures (e.g., trees, taxonomies) and power-law distributions with substantially fewer dimensions compared to Euclidean counterparts. Recent advances have leveraged these properties to enhance foundation models, including improving LLMs' complex reasoning ability, VLMs' zero-shot generalization, and cross-modal semantic alignment, while maintaining parameter efficiency. This paper provides a comprehensive review of hyperbolic neural networks and their recent development for foundation models. We further outline key challenges and research directions to advance the field.