# Hyperbolic Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2210.01542 · HTML: https://mlsanity.com/paper/2210.01542/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 63 of its 97 references that are in the map.
- First posted on arXiv 4 Oct 2022.

## Builds on (most cited first, 30 of 63)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels](https://mlsanity.com/paper/2004.13649/) (2020)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [Leveraging Procedural Generation to Benchmark Reinforcement Learning](https://mlsanity.com/paper/1912.01588/) (2019)
- [Reinforcement Learning with Augmented Data](https://mlsanity.com/paper/2004.14990/) (2020)
- [Towards Principled Methods for Training Generative Adversarial Networks](https://mlsanity.com/paper/1701.04862/) (2017)
- [Hyperbolic Neural Networks](https://mlsanity.com/paper/1805.09112/) (2018)
- [Quantifying Generalization in Reinforcement Learning](https://mlsanity.com/paper/1812.02341/) (2018)
- [Data-Efficient Reinforcement Learning with Self-Predictive Representations](https://mlsanity.com/paper/2007.05929/) (2020)
- [Challenges of Real-World Reinforcement Learning](https://mlsanity.com/paper/1904.12901/) (2019)
- [Improving Sample Efficiency in Model-Free Reinforcement Learning from Images](https://mlsanity.com/paper/1910.01741/) (2019)

## Built on by (most cited first, 25 of 25)

- [Hyperbolic Deep Learning in Computer Vision: A Survey](https://mlsanity.com/paper/2305.06611/) (2023)
- [EDGI: Equivariant Diffusion for Planning with Embodied Agents](https://mlsanity.com/paper/2303.12410/) (2023)
- [Hyperbolic Fine-Tuning for Large Language Models](https://mlsanity.com/paper/2410.04010/) (2024)
- [Hyperbolic Diffusion Embedding and Distance for Hierarchical Representation Learning](https://mlsanity.com/paper/2305.18962/) (2023)
- [Hyperbolic Sliced-Wasserstein via Geodesic and Horospherical Projections](https://mlsanity.com/paper/2211.10066/) (2022)
- [Embedding Geometries of Contrastive Language-Image Pre-Training](https://mlsanity.com/paper/2409.13079/) (2024)
- [GeoWorld: Geometric World Models](https://mlsanity.com/paper/2602.23058/) (2026)
- [Leveraging Hyperbolic Embeddings for Coarse-to-Fine Robot Design](https://mlsanity.com/paper/2311.00462/) (2023)
- [Stable Deep Reinforcement Learning via Isotropic Gaussian Representations](https://mlsanity.com/paper/2602.19373/) (2026)
- [Capacity Bounds for Hyperbolic Neural Network Representations of Latent Tree Structures](https://mlsanity.com/paper/2308.09250/) (2023)
- [Adversarial Attacks on Hyperbolic Networks](https://mlsanity.com/paper/2412.01495/) (2024)
- [A group-theoretic framework for machine learning in hyperbolic spaces](https://mlsanity.com/paper/2501.06934/) (2025)
- [Why are hyperbolic neural networks effective? A study on hierarchical representation capability](https://mlsanity.com/paper/2402.02478/) (2024)
- [Kuramoto Oscillators and Swarms on Manifolds for Geometry Informed Machine Learning](https://mlsanity.com/paper/2405.09453/) (2024)
- [Hyp2Nav: Hyperbolic Planning and Curiosity for Crowd Navigation](https://mlsanity.com/paper/2407.13567/) (2024)
- [Hyperbolic Residual Quantization: Discrete Representations for Data with Latent Hierarchies](https://mlsanity.com/paper/2505.12404/) (2025)
- [Surrogate-Assisted Evolutionary Reinforcement Learning Based on Autoencoder and Hyperbolic Neural Network](https://mlsanity.com/paper/2505.19423/) (2025)
- [Randomized HyperSteiner: A Stochastic Delaunay Triangulation Heuristic for the Hyperbolic Steiner Minimal Tree](https://mlsanity.com/paper/2510.09328/) (2025)
- [HLoOP -- Hyperbolic 2-space Local Outlier Probabilities](https://mlsanity.com/paper/2312.03895/) (2023)
- [A Bottom-Up Approach to Class-Agnostic Image Segmentation](https://mlsanity.com/paper/2409.13687/) (2024)
- [Uncovering RL Integration in SSL Loss: Objective-Specific Implications for Data-Efficient RL](https://mlsanity.com/paper/2410.17428/) (2024)
- [Free Random Projection for In-Context Reinforcement Learning](https://mlsanity.com/paper/2504.06983/) (2025)
- [Improved Offline Reinforcement Learning via Quantum Metric Encoding](https://mlsanity.com/paper/2511.10187/) (2025)
- [Curvature-Aware Captioning:Leveraging Geodesic Attention for 3D Scene Understanding](https://mlsanity.com/paper/2605.08808/) (2026)
- [Information-Geometric Optimization on Spheres](https://mlsanity.com/paper/2606.07588/) (2026)

## Abstract (arXiv)

We propose a new class of deep reinforcement learning (RL) algorithms that model latent representations in hyperbolic space. Sequential decision-making requires reasoning about the possible future consequences of current behavior. Consequently, capturing the relationship between key evolving features for a given task is conducive to recovering effective policies. To this end, hyperbolic geometry provides deep RL models with a natural basis to precisely encode this inherently hierarchical information. However, applying existing methodologies from the hyperbolic deep learning literature leads to fatal optimization instabilities due to the non-stationarity and variance characterizing RL gradient estimators. Hence, we design a new general method that counteracts such optimization challenges and enables stable end-to-end learning with deep hyperbolic representations. We empirically validate our framework by applying it to popular on-policy and off-policy RL algorithms on the Procgen and Atari 100K benchmarks, attaining near universal performance and generalization benefits. Given its natural fit, we hope future RL research will consider hyperbolic representations as a standard tool.