# A homotopy training algorithm for fully connected neural networks

arXiv: https://arxiv.org/abs/1903.09872 · HTML: https://mlsanity.com/paper/1903.09872/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Mar 2019.

## Built on by (most cited first, 7 of 7)

- [Homotopy-based training of NeuralODEs for accurate dynamics discovery](https://mlsanity.com/paper/2210.01407/) (2022)
- [Homotopy Relaxation Training Algorithms for Infinite-Width Two-Layer ReLU Neural Networks](https://mlsanity.com/paper/2309.15244/) (2023)
- [Deep Contrastive Graph Representation via Adaptive Homotopy Learning](https://mlsanity.com/paper/2106.09244/) (2021)
- [A Weight Initialization Based on the Linear Product Structure for Neural Networks](https://mlsanity.com/paper/2109.00125/) (2021)
- [Learn Singularly Perturbed Solutions via Homotopy Dynamics](https://mlsanity.com/paper/2502.00488/) (2025)
- [EuclidNets: An Alternative Operation for Efficient Inference of Deep Learning Models](https://mlsanity.com/paper/2212.11803/) (2022)
- [HomOpt: A Homotopy-Based Hyperparameter Optimization Method](https://mlsanity.com/paper/2308.03317/) (2023)

## Abstract (arXiv)

In this paper, we present a Homotopy Training Algorithm (HTA) to solve optimization problems arising from fully connected neural networks with complicated structures. The HTA dynamically builds the neural network starting from a simplified version to the fully connected network via adding layers and nodes adaptively. Therefore, the corresponding optimization problem is easy to solve at the beginning and connects to the original model via a continuous path guided by the HTA, which provides a high probability to get a global minimum. By gradually increasing the complexity of model along the continuous path, the HTA gets a rather good solution to the original loss function. This is confirmed by various numerical results including VGG models on CIFAR-10. For example, on the VGG13 model with batch normalization, HTA reduces the error rate by 11.86% on test dataset comparing with the traditional method. Moreover, the HTA also allows us to find the optimal structure for a fully connected neural network by building the neutral network adaptively.