# Strongly-Typed Recurrent Neural Networks

arXiv: https://arxiv.org/abs/1602.02218 · HTML: https://mlsanity.com/paper/1602.02218/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Feb 2016.

## Built on by (most cited first, 13 of 13)

- [Hierarchically Gated Recurrent Neural Network for Sequence Modeling](https://mlsanity.com/paper/2311.04823/) (2023)
- [Parallelizing Legendre Memory Unit Training](https://mlsanity.com/paper/2102.11417/) (2021)
- [MetaLA: Unified Optimal Linear Approximation to Softmax Attention Map](https://mlsanity.com/paper/2411.10741/) (2024)
- [Sequential-Parallel Duality in Prefix Scannable Models](https://mlsanity.com/paper/2506.10918/) (2025)
- [Fast weight programming and linear transformers: from machine learning to neurobiology](https://mlsanity.com/paper/2508.08435/) (2025)
- [Why Are Positional Encodings Nonessential for Deep Autoregressive Transformers? Revisiting a Petroglyph](https://mlsanity.com/paper/2501.00659/) (2024)
- [Fusing Structure from Motion and Simulation-Augmented Pose Regression from Optical Flow for Challenging Indoor Environments](https://mlsanity.com/paper/2304.07250/) (2023)
- [Resona: Improving Context Copying in Linear Recurrence Models with Retrieval](https://mlsanity.com/paper/2503.22913/) (2025)
- [RecurFormer: Not All Transformer Heads Need Self-Attention](https://mlsanity.com/paper/2410.12850/) (2024)
- [Birdie: Advancing State Space Models with Reward-Driven Objectives and Curricula](https://mlsanity.com/paper/2411.01030/) (2024)
- [Encoding architecture algebra](https://mlsanity.com/paper/2410.11776/) (2024)
- [Bidirectional Linear Recurrent Models for Sequence-Level Multisource Fusion](https://mlsanity.com/paper/2504.08964/) (2025)
- [Gradland: On Phenomenal Experience, Differentiated Across Many Dimensions](https://mlsanity.com/paper/2609.09306/) (2026)

## Abstract (arXiv)

Recurrent neural networks are increasing popular models for sequential learning. Unfortunately, although the most effective RNN architectures are perhaps excessively complicated, extensive searches have not found simpler alternatives. This paper imports ideas from physics and functional programming into RNN design to provide guiding principles. From physics, we introduce type constraints, analogous to the constraints that forbids adding meters to seconds. From functional programming, we require that strongly-typed architectures factorize into stateless learnware and state-dependent firmware, reducing the impact of side-effects. The features learned by strongly-typed nets have a simple semantic interpretation via dynamic average-pooling on one-dimensional convolutions. We also show that strongly-typed gradients are better behaved than in classical architectures, and characterize the representational power of strongly-typed nets. Finally, experiments show that, despite being more constrained, strongly-typed architectures achieve lower training and comparable generalization error to classical architectures.