# Input-Output Equivalence of Unitary and Contractive RNNs

arXiv: https://arxiv.org/abs/1910.13672 · HTML: https://mlsanity.com/paper/1910.13672/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Oct 2019.

## Built on by (most cited first, 3 of 3)

- [Implicit Bias of Linear RNNs](https://mlsanity.com/paper/2101.07833/) (2021)
- [Governance-Aware Hybrid Fine-Tuning for Multilingual Large Language Models](https://mlsanity.com/paper/2512.17344/) (2025)
- [Hybrid and Unitary PEFT for Resource-Efficient Large Language Models](https://mlsanity.com/paper/2507.18076/) (2025)

## Abstract (arXiv)

Unitary recurrent neural networks (URNNs) have been proposed as a method to overcome the vanishing and exploding gradient problem in modeling data with long-term dependencies. A basic question is how restrictive is the unitary constraint on the possible input-output mappings of such a network? This work shows that for any contractive RNN with ReLU activations, there is a URNN with at most twice the number of hidden states and the identical input-output mapping. Hence, with ReLU activations, URNNs are as expressive as general RNNs. In contrast, for certain smooth activations, it is shown that the input-output mapping of an RNN cannot be matched with a URNN, even with an arbitrary number of states. The theoretical results are supported by experiments on modeling of slowly-varying dynamical systems.