# Towards Scalable Hyperbolic Neural Networks using Taylor Series Approximations

arXiv: https://arxiv.org/abs/2206.03610 · HTML: https://mlsanity.com/paper/2206.03610/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 51 references that are in the map.
- First posted on arXiv 7 Jun 2022.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Geometric deep learning: going beyond Euclidean data](https://mlsanity.com/paper/1611.08097/) (2016)
- [Convolutional 2D Knowledge Graph Embeddings](https://mlsanity.com/paper/1707.01476/) (2017)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [Hyperbolic Neural Networks](https://mlsanity.com/paper/1805.09112/) (2018)
- [Hyperbolic Graph Convolutional Neural Networks](https://mlsanity.com/paper/1910.12933/) (2019)
- [Hyperbolic Image Embeddings](https://mlsanity.com/paper/1904.02239/) (2019)
- [Riemannian Adaptive Optimization Methods](https://mlsanity.com/paper/1810.00760/) (2018)
- [Hyperbolic Attention Networks](https://mlsanity.com/paper/1805.09786/) (2018)
- [Continuous Hierarchical Representations with Poincar'e Variational Auto-Encoders](https://mlsanity.com/paper/1901.06033/) (2019)
- [Constant Curvature Graph Convolutional Networks](https://mlsanity.com/paper/1911.05076/) (2019)
- [Fully Hyperbolic Neural Networks](https://mlsanity.com/paper/2105.14686/) (2021)
- [Hyperbolic Graph Attention Network](https://mlsanity.com/paper/1912.03046/) (2019)
- [Lorentzian Graph Convolutional Networks](https://mlsanity.com/paper/2104.07477/) (2021)
- [Few-Shot Learning Through an Information Retrieval Lens](https://mlsanity.com/paper/1707.02610/) (2017)
- [Hyperbolic Variational Graph Neural Network for Modeling Dynamic Graphs](https://mlsanity.com/paper/2104.02228/) (2021)
- [Self-Supervised Hyperboloid Representations from Logical Queries over Knowledge Graphs](https://mlsanity.com/paper/2012.13023/) (2020)

## Built on by (most cited first, 3 of 3)

- [Hyperbolic Deep Learning in Computer Vision: A Survey](https://mlsanity.com/paper/2305.06611/) (2023)
- [DeepHGCN: Toward Deeper Hyperbolic Graph Convolutional Networks](https://mlsanity.com/paper/2310.02027/) (2023)
- [Continuous Geometry-Aware Graph Diffusion via Hyperbolic Neural PDE](https://mlsanity.com/paper/2406.01282/) (2024)

## Abstract (arXiv)

Hyperbolic networks have shown prominent improvements over their Euclidean counterparts in several areas involving hierarchical datasets in various domains such as computer vision, graph analysis, and natural language processing. However, their adoption in practice remains restricted due to (i) non-scalability on accelerated deep learning hardware, (ii) vanishing gradients due to the closure of hyperbolic space, and (iii) information loss due to frequent mapping between local tangent space and fully hyperbolic space. To tackle these issues, we propose the approximation of hyperbolic operators using Taylor series expansions, which allows us to reformulate the computationally expensive tangent and cosine hyperbolic functions into their polynomial equivariants which are more efficient. This allows us to retain the benefits of preserving the hierarchical anatomy of the hyperbolic space, while maintaining the scalability over current accelerated deep learning infrastructure. The polynomial formulation also enables us to utilize the advancements in Euclidean networks such as gradient clipping and ReLU activation to avoid vanishing gradients and remove errors due to frequent switching between tangent space and hyperbolic space. Our empirical evaluation on standard benchmarks in the domain of graph analysis and computer vision shows that our polynomial formulation is as scalable as Euclidean architectures, both in terms of memory and time complexity, while providing results as effective as hyperbolic models. Moreover, our formulation also shows a considerable improvement over its baselines due to our solution to vanishing gradients and information loss.