# Contextual Recurrent Neural Networks

arXiv: https://arxiv.org/abs/1902.03455 · HTML: https://mlsanity.com/paper/1902.03455/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Feb 2019.

## Built on by (most cited first, 5 of 5)

- [Residual Context Diffusion Language Models](https://mlsanity.com/paper/2601.22954/) (2026)
- [Fourier-RNNs for Modelling Noisy Physics Data](https://mlsanity.com/paper/2302.06534/) (2023)
- [Adaptable Symbolic Music Infilling with MIDI-RWKV](https://mlsanity.com/paper/2506.13001/) (2025)
- [Am I fit for this physical activity? Neural embedding of physical conditioning from inertial sensors](https://mlsanity.com/paper/2103.12095/) (2021)
- [Explicit Context Integrated Recurrent Neural Network for Sensor Data Applications](https://mlsanity.com/paper/2301.05031/) (2023)

## Abstract (arXiv)

There is an implicit assumption that by unfolding recurrent neural networks (RNN) in finite time, the misspecification of choosing a zero value for the initial hidden state is mitigated by later time steps. This assumption has been shown to work in practice and alternative initialization may be suggested but often overlooked. In this paper, we propose a method of parameterizing the initial hidden state of an RNN. The resulting architecture, referred to as a Contextual RNN, can be trained end-to-end. The performance on an associative retrieval task is found to improve by conditioning the RNN initial hidden state on contextual information from the input sequence. Furthermore, we propose a novel method of conditionally generating sequences using the hidden state parameterization of Contextual RNN.