# Hessian-free Optimization for Learning Deep Multidimensional Recurrent Neural Networks

arXiv: https://arxiv.org/abs/1509.03475 · HTML: https://mlsanity.com/paper/1509.03475/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Sep 2015.

## Built on by (most cited first, 5 of 5)

- [A New Perspective on Shampoo's Preconditioner](https://mlsanity.com/paper/2406.17748/) (2024)
- [The Potential of Second-Order Optimization for LLMs: A Study with Full Gauss-Newton](https://mlsanity.com/paper/2510.09378/) (2025)
- [Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptation](https://mlsanity.com/paper/2510.00192/) (2025)
- [Signal to Noise Ratio Loss Function](https://mlsanity.com/paper/2110.12275/) (2021)
- [Accelerated Training of Federated Learning via Second-Order Methods](https://mlsanity.com/paper/2505.23588/) (2025)

## Abstract (arXiv)

Multidimensional recurrent neural networks (MDRNNs) have shown a remarkable performance in the area of speech and handwriting recognition. The performance of an MDRNN is improved by further increasing its depth, and the difficulty of learning the deeper network is overcome by using Hessian-free (HF) optimization. Given that connectionist temporal classification (CTC) is utilized as an objective of learning an MDRNN for sequence labeling, the non-convexity of CTC poses a problem when applying HF to the network. As a solution, a convex approximation of CTC is formulated and its relationship with the EM algorithm and the Fisher information matrix is discussed. An MDRNN up to a depth of 15 layers is successfully trained using HF, resulting in an improved performance for sequence labeling.