# Covariant Gradient Descent

arXiv: https://arxiv.org/abs/2504.05279 · HTML: https://mlsanity.com/paper/2504.05279/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 25 references that are in the map.
- First posted on arXiv 7 Apr 2025.

## Builds on (most cited first, 11 of 11)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients](https://mlsanity.com/paper/2010.07468/) (2020)
- [The world as a neural network](https://mlsanity.com/paper/2008.01540/) (2020)
- [Towards a theory of machine learning](https://mlsanity.com/paper/2004.09280/) (2020)
- [Towards a theory of quantum gravity from neural networks](https://mlsanity.com/paper/2111.00903/) (2021)
- [The Autodidactic Universe](https://mlsanity.com/paper/2104.03902/) (2021)
- [Dataset-learning duality and emergent criticality](https://mlsanity.com/paper/2405.17391/) (2024)
- [Emergent field theories from neural networks](https://mlsanity.com/paper/2411.08138/) (2024)
- [Autonomous particles](https://mlsanity.com/paper/2301.10077/) (2023)

## Built on by (most cited first, 3 of 3)

- [Geometric Learning Dynamics](https://mlsanity.com/paper/2504.14728/) (2025)
- [Molecular Learning Dynamics](https://mlsanity.com/paper/2504.10560/) (2025)
- [Geometric framework for biological evolution](https://mlsanity.com/paper/2603.15198/) (2026)

## Abstract (arXiv)

We present a manifestly covariant formulation of the gradient descent method, ensuring consistency across arbitrary coordinate systems and general curved trainable spaces. The optimization dynamics is defined using a covariant force vector and a covariant metric tensor, both computed from the first and second statistical moments of the gradients. These moments are estimated through time-averaging with an exponential weight function, which preserves linear computational complexity. We show that commonly used optimization methods such as RMSProp, Adam and AdaBelief correspond to special limits of the covariant gradient descent (CGD) and demonstrate how these methods can be further generalized and improved.