# Analyzing and Exploiting NARX Recurrent Neural Networks for Long-Term Dependencies

arXiv: https://arxiv.org/abs/1702.07805 · HTML: https://mlsanity.com/paper/1702.07805/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Feb 2017.

## Built on by (most cited first, 2 of 2)

- [Recurrence along Depth: Deep Convolutional Neural Networks with Recurrent Layer Aggregation](https://mlsanity.com/paper/2110.11852/) (2021)
- [Reinforcement Learning Controllers for Soft Robots using Learned Environments](https://mlsanity.com/paper/2410.18519/) (2024)

## Abstract (arXiv)

Recurrent neural networks (RNNs) have achieved state-of-the-art performance on many diverse tasks, from machine translation to surgical activity recognition, yet training RNNs to capture long-term dependencies remains difficult. To date, the vast majority of successful RNN architectures alleviate this problem using nearly-additive connections between states, as introduced by long short-term memory (LSTM). We take an orthogonal approach and introduce MIST RNNs, a NARX RNN architecture that allows direct connections from the very distant past. We show that MIST RNNs 1) exhibit superior vanishing-gradient properties in comparison to LSTM and previously-proposed NARX RNNs; 2) are far more efficient than previously-proposed NARX RNN architectures, requiring even fewer computations than LSTM; and 3) improve performance substantially over LSTM and Clockwork RNNs on tasks requiring very long-term dependencies.