# The unreasonable effectiveness of the forget gate

arXiv: https://arxiv.org/abs/1804.04849 · HTML: https://mlsanity.com/paper/1804.04849/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Apr 2018.

## Built on by (most cited first, 30 of 34)

- [Gated Delta Networks: Improving Mamba2 with Delta Rule](https://mlsanity.com/paper/2412.06464/) (2024)
- [Gated Linear Attention Transformers with Hardware-Efficient Training](https://mlsanity.com/paper/2312.06635/) (2023)
- [Titans: Learning to Memorize at Test Time](https://mlsanity.com/paper/2501.00663/) (2024)
- [HGRN2: Gated Linear RNNs with State Expansion](https://mlsanity.com/paper/2404.07904/) (2024)
- [Hierarchically Gated Recurrent Neural Network for Sequence Modeling](https://mlsanity.com/paper/2311.04823/) (2023)
- [Test-time regression: a unifying framework for designing sequence models with associative memory](https://mlsanity.com/paper/2501.12352/) (2025)
- [A Comprehensive Survey on Hardware-Aware Neural Architecture Search](https://mlsanity.com/paper/2101.09336/) (2021)
- [Forgetting Transformer: Softmax Attention with a Forget Gate](https://mlsanity.com/paper/2503.02130/) (2025)
- [MesaNet: Sequence Modeling by Locally Optimal Test-Time Training](https://mlsanity.com/paper/2506.05233/) (2025)
- [Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search](https://mlsanity.com/paper/2508.15884/) (2025)
- [Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing](https://mlsanity.com/paper/2501.00658/) (2024)
- [HiCache: A Plug-in Scaled-Hermite Upgrade for Taylor-Style Cache-then-Forecast Diffusion Acceleration](https://mlsanity.com/paper/2508.16984/) (2025)
- [How Does Momentum Benefit Deep Neural Networks Architecture Design? A Few Case Studies](https://mlsanity.com/paper/2110.07034/) (2021)
- [Slower is Better: Revisiting the Forgetting Mechanism in LSTM for Slower Information Decay](https://mlsanity.com/paper/2105.05944/) (2021)
- [OpenDPD: An Open-Source End-to-End Learning & Benchmarking Framework for Wideband Power Amplifier Modeling and Digital Pre-Distortion](https://mlsanity.com/paper/2401.08318/) (2024)
- [Rodimus*: Breaking the Accuracy-Efficiency Trade-Off with Efficient Attentions](https://mlsanity.com/paper/2410.06577/) (2024)
- [Recurrent Neural Networks for Learning Long-term Temporal Dependencies with Reanalysis of Time Scale Representation](https://mlsanity.com/paper/2111.03282/) (2021)
- [Warming up recurrent neural networks to maximise reachable multistability greatly improves learning](https://mlsanity.com/paper/2106.01001/) (2021)
- [A Brief Survey of Associations Between Meta-Learning and General AI](https://mlsanity.com/paper/2101.04283/) (2021)
- [Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions](https://mlsanity.com/paper/2108.13055/) (2021)
- [Oscillatory Fourier Neural Network: A Compact and Efficient Architecture for Sequential Processing](https://mlsanity.com/paper/2109.13090/) (2021)
- [Time-Scale Coupling Between States and Parameters in Recurrent Neural Networks](https://mlsanity.com/paper/2508.12121/) (2025)
- [Gates Are Not What You Need in RNNs](https://mlsanity.com/paper/2108.00527/) (2021)
- [Machine Learning based prediction of Glucose Levels in Type 1 Diabetes Patients with the use of Continuous Glucose Monitoring Data](https://mlsanity.com/paper/2302.12856/) (2023)
- [DeltaDPD: Exploiting Dynamic Temporal Sparsity in Recurrent Neural Networks for Energy-Efficient Wideband Digital Predistortion](https://mlsanity.com/paper/2505.06250/) (2025)
- [Don't Pay Attention](https://mlsanity.com/paper/2506.11305/) (2025)
- [JaneEye: A 12-nm 2K-FPS 18.9-μJ/Frame Event-based Eye Tracking Accelerator](https://mlsanity.com/paper/2510.01213/) (2025)
- [Learning to Forget: Continual Learning with Adaptive Weight Decay](https://mlsanity.com/paper/2604.27063/) (2026)
- [Learning Human Cognitive Appraisal Through Reinforcement Memory Unit](https://mlsanity.com/paper/2208.03473/) (2022)
- [Fast Saturating Gate for Learning Long Time Scales with Recurrent Neural Networks](https://mlsanity.com/paper/2210.01348/) (2022)

## Abstract (arXiv)

Given the success of the gated recurrent unit, a natural question is whether all the gates of the long short-term memory (LSTM) network are necessary. Previous research has shown that the forget gate is one of the most important gates in the LSTM. Here we show that a forget-gate-only version of the LSTM with chrono-initialized biases, not only provides computational savings but outperforms the standard LSTM on multiple benchmark datasets and competes with some of the best contemporary models. Our proposed network, the JANET, achieves accuracies of 99% and 92.5% on the MNIST and pMNIST datasets, outperforming the standard LSTM which yields accuracies of 98.5% and 91%.