# Surprisal-Driven Feedback in Recurrent Networks

arXiv: https://arxiv.org/abs/1608.06027 · HTML: https://mlsanity.com/paper/1608.06027/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Aug 2016.

## Built on by (most cited first, 2 of 2)

- [Training and Generating Neural Networks in Compressed Weight Space](https://mlsanity.com/paper/2112.15545/) (2021)
- [Never Reset Again: A Mathematical Framework for Continual Inference in Recurrent Neural Networks](https://mlsanity.com/paper/2412.15983/) (2024)

## Abstract (arXiv)

Recurrent neural nets are widely used for predicting temporal data. Their inherent deep feedforward structure allows learning complex sequential patterns. It is believed that top-down feedback might be an important missing ingredient which in theory could help disambiguate similar patterns depending on broader context. In this paper we introduce surprisal-driven recurrent networks, which take into account past error information when making new predictions. This is achieved by continuously monitoring the discrepancy between most recent predictions and the actual observations. Furthermore, we show that it outperforms other stochastic and fully deterministic approaches on enwik8 character level prediction task achieving 1.37 BPC on the test portion of the text.