# Combining Natural Gradient with Hessian Free Methods for Sequence Training

arXiv: https://arxiv.org/abs/1810.01873 · HTML: https://mlsanity.com/paper/1810.01873/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Oct 2018.

## Built on by (most cited first, 2 of 2)

- [A Distributed Optimisation Framework Combining Natural Gradient with Hessian-Free for Discriminative Sequence Training](https://mlsanity.com/paper/2103.07554/) (2021)
- [A Treatise On FST Lattice Based MMI Training](https://mlsanity.com/paper/2210.08918/) (2022)

## Abstract (arXiv)

This paper presents a new optimisation approach to train Deep Neural Networks (DNNs) with discriminative sequence criteria. At each iteration, the method combines information from the Natural Gradient (NG) direction with local curvature information of the error surface that enables better paths on the parameter manifold to be traversed. The method is derived using an alternative derivation of Taylor's theorem using the concepts of manifolds, tangent vectors and directional derivatives from the perspective of Information Geometry. The efficacy of the method is shown within a Hessian Free (HF) style optimisation framework to sequence train both standard fully-connected DNNs and Time Delay Neural Networks as speech recognition acoustic models. It is shown that for the same number of updates the proposed approach achieves larger reductions in the word error rate (WER) than both NG and HF, and also leads to a lower WER than standard stochastic gradient descent. The paper also addresses the issue of over-fitting due to mismatch between training criterion and Word Error Rate (WER) that primarily arises during sequence training of ReLU-DNN models.