# A Distributed Optimisation Framework Combining Natural Gradient with Hessian-Free for Discriminative Sequence Training

arXiv: https://arxiv.org/abs/2103.07554 · HTML: https://mlsanity.com/paper/2103.07554/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 99 references that are in the map.
- First posted on arXiv 12 Mar 2021.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [State-of-the-art Speech Recognition With Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01769/) (2017)
- [Revisiting Natural Gradient for Deep Networks](https://mlsanity.com/paper/1301.3584/) (2013)
- [Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis](https://mlsanity.com/paper/1806.03884/) (2018)
- [RWTH ASR Systems for LibriSpeech: Hybrid vs Attention -- w/o Data Augmentation](https://mlsanity.com/paper/1905.03072/) (2019)
- [Natural Neural Networks](https://mlsanity.com/paper/1507.00210/) (2015)
- [Second-Order Optimization for Non-Convex Machine Learning: An Empirical Study](https://mlsanity.com/paper/1708.07827/) (2017)
- [Optimizing expected word error rate via sampling for speech recognition](https://mlsanity.com/paper/1706.02776/) (2017)
- [Parallel training of DNNs with Natural Gradient and Parameter Averaging](https://mlsanity.com/paper/1410.7455/) (2014)
- [The Microsoft 2016 Conversational Speech Recognition System](https://mlsanity.com/paper/1609.03528/) (2016)
- [Improved TDNNs using Deep Kernels and Frequency Dependent Grid-RNNs](https://mlsanity.com/paper/1802.06412/) (2018)
- [A Highly Efficient Distributed Deep Learning System For Automatic Speech Recognition](https://mlsanity.com/paper/1907.05701/) (2019)
- [The IBM 2016 English Conversational Telephone Speech Recognition System](https://mlsanity.com/paper/1604.08242/) (2016)
- [Distributed Deep Learning Strategies For Automatic Speech Recognition](https://mlsanity.com/paper/1904.04956/) (2019)
- [Accelerating Hessian-free optimization for deep neural networks by implicit preconditioning and sampling](https://mlsanity.com/paper/1309.1508/) (2013)
- [Sequence Training of DNN Acoustic Models With Natural Gradient](https://mlsanity.com/paper/1804.02204/) (2018)
- [Combining Natural Gradient with Hessian Free Methods for Sequence Training](https://mlsanity.com/paper/1810.01873/) (2018)
- [A Common Framework for Natural Gradient and Taylor based Optimisation using Manifold Theory](https://mlsanity.com/paper/1803.09791/) (2018)

## Built on by (most cited first, 1 of 1)

- [Asynchronous Decentralized Distributed Training of Acoustic Models](https://mlsanity.com/paper/2110.11199/) (2021)

## Abstract (arXiv)

This paper presents a novel natural gradient and Hessian-free (NGHF) optimisation framework for neural network training that can operate efficiently in a distributed manner. It relies on the linear conjugate gradient (CG) algorithm to combine the natural gradient (NG) method with local curvature information from Hessian-free (HF) or other second-order methods. A solution to a numerical issue in CG allows effective parameter updates to be generated with far fewer CG iterations than usually used (e.g. 5-8 instead of 200). This work also presents a novel preconditioning approach to improve the progress made by individual CG iterations for models with shared parameters. Although applicable to other training losses and model structures, NGHF is investigated in this paper for lattice-based discriminative sequence training for hybrid hidden Markov model acoustic models using a standard recurrent neural network, long short-term memory, and time delay neural network models for output probability calculation. Automatic speech recognition experiments are reported on the multi-genre broadcast data set for a range of different acoustic model types. These experiments show that NGHF achieves larger word error rate reductions than standard stochastic gradient descent or Adam, while requiring orders of magnitude fewer parameter updates.