# Training of Deep Neural Networks based on Distance Measures using RMSProp

arXiv: https://arxiv.org/abs/1708.01911 · HTML: https://mlsanity.com/paper/1708.01911/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Aug 2017.

## Built on by (most cited first, 8 of 8)

- [QKAN-LSTM: Quantum-inspired Kolmogorov-Arnold Long Short-term Memory](https://mlsanity.com/paper/2512.05049/) (2025)
- [A spatial-temporal short-term traffic flow prediction model based on dynamical-learning graph convolution mechanism](https://mlsanity.com/paper/2205.04762/) (2022)
- [Towards Simple and Provable Parameter-Free Adaptive Gradient Methods](https://mlsanity.com/paper/2412.19444/) (2024)
- [A Solver + Gradient Descent Training Algorithm for Deep Neural Networks](https://mlsanity.com/paper/2207.03264/) (2022)
- [Resource-Efficient Glioma Segmentation on Sub-Saharan MRI](https://mlsanity.com/paper/2509.09469/) (2025)
- [A Deep Learning-based Method to Extract Lumen and Media-Adventitia in Intravascular Ultrasound Images](https://mlsanity.com/paper/2102.10480/) (2021)
- [Benchmarking YOLOv8 for Optimal Crack Detection in Civil Infrastructure](https://mlsanity.com/paper/2501.06922/) (2025)
- [Distribution of hitting times for dissipative random dynamical systems on ℝd, with application to stochastic gradient descent](https://mlsanity.com/paper/2609.30274/) (2026)

## Abstract (arXiv)

The vanishing gradient problem was a major obstacle for the success of deep learning. In recent years it was gradually alleviated through multiple different techniques. However the problem was not really overcome in a fundamental way, since it is inherent to neural networks with activation functions based on dot products. In a series of papers, we are going to analyze alternative neural network structures which are not based on dot products. In this first paper, we revisit neural networks built up of layers based on distance measures and Gaussian activation functions. These kinds of networks were only sparsely used in the past since they are hard to train when using plain stochastic gradient descent methods. We show that by using Root Mean Square Propagation (RMSProp) it is possible to efficiently learn multi-layer neural networks. Furthermore we show that when appropriately initialized these kinds of neural networks suffer much less from the vanishing and exploding gradient problem than traditional neural networks even for deep networks.