# Feedforward and Recurrent Neural Networks Backward Propagation and Hessian in Matrix Form

arXiv: https://arxiv.org/abs/1709.06080 · HTML: https://mlsanity.com/paper/1709.06080/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Sep 2017.

## Built on by (most cited first, 3 of 3)

- [Batch Normalization Preconditioning for Neural Network Training](https://mlsanity.com/paper/2108.01110/) (2021)
- [Derivation of Back-propagation for Graph Convolutional Networks using Matrix Calculus and its Application to Explainable Artificial Intelligence](https://mlsanity.com/paper/2408.01408/) (2024)
- [Scalable Kronecker-Fisher Approximation: Efficient Hessian Analysis for Billion-Parameter Language Models Compression](https://mlsanity.com/paper/2609.02451/) (2026)

## Abstract (arXiv)

In this paper we focus on the linear algebra theory behind feedforward (FNN) and recurrent (RNN) neural networks. We review backward propagation, including backward propagation through time (BPTT). Also, we obtain a new exact expression for Hessian, which represents second order effects. We show that for t time steps the weight gradient can be expressed as a rank-t matrix, while the weight Hessian is as a sum of t2 Kronecker products of rank-1 and WTAW matrices, for some matrix A and weight matrix W. Also, we show that for a mini-batch of size r, the weight update can be expressed as a rank-rt matrix. Finally, we briefly comment on the eigenvalues of the Hessian matrix.