# A geometric interpretation of stochastic gradient descent using diffusion metrics

arXiv: https://arxiv.org/abs/1910.12194 · HTML: https://mlsanity.com/paper/1910.12194/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Oct 2019.

## Built on by (most cited first, 1 of 1)

- [Geometric Deep Learning: a Temperature Based Analysis of Graph Neural Networks](https://mlsanity.com/paper/2309.00699/) (2023)

## Abstract (arXiv)

Stochastic gradient descent (SGD) is a key ingredient in the training of deep neural networks and yet its geometrical significance appears elusive. We study a deterministic model in which the trajectories of our dynamical systems are described via geodesics of a family of metrics arising from the diffusion matrix. These metrics encode information about the highly non-isotropic gradient noise in SGD. We establish a parallel with General Relativity models, where the role of the electromagnetic field is played by the gradient of the loss function. We compute an example of a two layer network.