# Gated Orthogonal Recurrent Units: On Learning to Forget

arXiv: https://arxiv.org/abs/1706.02761 · HTML: https://mlsanity.com/paper/1706.02761/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jun 2017.

## Built on by (most cited first, 30 of 31)

- [Parallelizing Linear Transformers with the Delta Rule over Sequence Length](https://mlsanity.com/paper/2406.06484/) (2024)
- [Liquid Structural State-Space Models](https://mlsanity.com/paper/2209.12951/) (2022)
- [A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?](https://mlsanity.com/paper/2303.11717/) (2023)
- [On the difficulty of learning chaotic dynamics with RNNs](https://mlsanity.com/paper/2110.07238/) (2021)
- [Medical-VLBERT: Medical Visual Language BERT for COVID-19 CT Report Generation With Alternate Learning](https://mlsanity.com/paper/2108.05067/) (2021)
- [A Survey of Geometric Optimization for Deep Learning: From Euclidean Space to Riemannian Manifold](https://mlsanity.com/paper/2302.08210/) (2023)
- [Mano: Restriking Manifold Optimization for LLM Training](https://mlsanity.com/paper/2601.23000/) (2026)
- [Working Memory Connections for LSTM](https://mlsanity.com/paper/2109.00020/) (2021)
- [Learning to Dissipate Energy in Oscillatory State-Space Models](https://mlsanity.com/paper/2505.12171/) (2025)
- [Slower is Better: Revisiting the Forgetting Mechanism in LSTM for Slower Information Decay](https://mlsanity.com/paper/2105.05944/) (2021)
- [On the Power of Convolution Augmented Transformer](https://mlsanity.com/paper/2407.05591/) (2024)
- [What is the best RNN-cell structure to forecast each time series behavior?](https://mlsanity.com/paper/2203.07844/) (2022)
- [Evaluation of Question Answering Systems: Complexity of judging a natural language](https://mlsanity.com/paper/2209.12617/) (2022)
- [Positional Encoding Helps Recurrent Neural Networks Handle a Large Vocabulary](https://mlsanity.com/paper/2402.00236/) (2024)
- [Recurrent Neural Networks for Learning Long-term Temporal Dependencies with Reanalysis of Time Scale Representation](https://mlsanity.com/paper/2111.03282/) (2021)
- [Orthogonal Gated Recurrent Unit with Neumann-Cayley Transformation](https://mlsanity.com/paper/2208.06496/) (2022)
- [Emergence of the Primacy Effect in Structured State-Space Models](https://mlsanity.com/paper/2502.13729/) (2025)
- [Warming up recurrent neural networks to maximise reachable multistability greatly improves learning](https://mlsanity.com/paper/2106.01001/) (2021)
- [Revealing and Mitigating the Local Pattern Shortcuts of Mamba](https://mlsanity.com/paper/2410.15678/) (2024)
- [RotLSTM: Rotating Memories in Recurrent Neural Networks](https://mlsanity.com/paper/2105.00357/) (2021)
- [Comparative Evaluation of Metaheuristic Algorithms for Hyperparameter Selection in Short-Term Weather Forecasting](https://mlsanity.com/paper/2309.02600/) (2023)
- [Separation capacity of linear reservoirs with random connectivity matrix](https://mlsanity.com/paper/2404.17429/) (2024)
- [Machine Learning in Short-Reach Optical Systems: A Comprehensive Survey](https://mlsanity.com/paper/2405.09557/) (2024)
- [Hybrid Quantum-Classical Recurrent Neural Networks](https://mlsanity.com/paper/2510.25557/) (2025)
- [KOSS: Kalman-Optimal Selective State Spaces for Long-Term Sequence Modeling](https://mlsanity.com/paper/2512.16723/) (2025)
- [Gated recurrent units and temporal convolutional network for multilabel classification](https://mlsanity.com/paper/2110.04414/) (2021)
- [Breaking Time Invariance: Assorted-Time Normalization for RNNs](https://mlsanity.com/paper/2209.14439/) (2022)
- [Compact Recurrent Transformer with Persistent Memory](https://mlsanity.com/paper/2505.00929/) (2025)
- [mGRADE: Minimal Recurrent Gating Meets Delay Convolutions for Lightweight Sequence Modeling](https://mlsanity.com/paper/2507.01829/) (2025)
- [ManifoldFlow: Spectrum-Learnable Stiefel Optimization via SPD Relaxation](https://mlsanity.com/paper/2607.04535/) (2026)

## Abstract (arXiv)

We present a novel recurrent neural network (RNN) based model that combines the remembering ability of unitary RNNs with the ability of gated RNNs to effectively forget redundant/irrelevant information in its memory. We achieve this by extending unitary RNNs with a gating mechanism. Our model is able to outperform LSTMs, GRUs and Unitary RNNs on several long-term dependency benchmark tasks. We empirically both show the orthogonal/unitary RNNs lack the ability to forget and also the ability of GORU to simultaneously remember long term dependencies while forgetting irrelevant information. This plays an important role in recurrent neural networks. We provide competitive results along with an analysis of our model on many natural sequential tasks including the bAbI Question Answering, TIMIT speech spectrum prediction, Penn TreeBank, and synthetic tasks that involve long-term dependencies such as algorithmic, parenthesis, denoising and copying tasks.