# EA-CG: An Approximate Second-Order Method for Training Fully-Connected Neural Networks

arXiv: https://arxiv.org/abs/1802.06502 · HTML: https://mlsanity.com/paper/1802.06502/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Feb 2018.

## Built on by (most cited first, 1 of 1)

- [Tractable structured natural gradient descent using local parameterizations](https://mlsanity.com/paper/2102.07405/) (2021)

## Abstract (arXiv)

For training fully-connected neural networks (FCNNs), we propose a practical approximate second-order method including: 1) an approximation of the Hessian matrix and 2) a conjugate gradient (CG) based method. Our proposed approximate Hessian matrix is memory-efficient and can be applied to any FCNNs where the activation and criterion functions are twice differentiable. We devise a CG-based method incorporating one-rank approximation to derive Newton directions for training FCNNs, which significantly reduces both space and time complexity. This CG-based method can be employed to solve any linear equation where the coefficient matrix is Kronecker-factored, symmetric and positive definite. Empirical studies show the efficacy and efficiency of our proposed method.