mlsanityData to 7 Oct 2026

The unreasonable effectiveness of the forget gate

arXiv 1804.04849 ↗First posted 13 Apr 2018cs.NE cs.LG stat.MLcited by 32

Not on the map. Why?

Abstract (arXiv)

Given the success of the gated recurrent unit, a natural question is whether all the gates of the long short-term memory (LSTM) network are necessary. Previous research has shown that the forget gate is one of the most important gates in the LSTM. Here we show that a forget-gate-only version of the LSTM with chrono-initialized biases, not only provides computational savings but outperforms the standard LSTM on multiple benchmark datasets and competes with some of the best contemporary models. Our proposed network, the JANET, achieves accuracies of 99% and 92.5% on the MNIST and pMNIST datasets, outperforming the standard LSTM which yields accuracies of 98.5% and 91%.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Apr 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 3; 2023: 3; 2024: 6; 2025: 9; 2026: 2.

32 papers.

Sort:

By area:

  1. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  2. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  3. 2024Titans: Learning to Memorize at Test Timecited by 303
  4. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 122
  5. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 100
  6. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 63
  7. 2021A Comprehensive Survey on Hardware-Aware Neural Architecture Searchcited by 60
  8. 2025Forgetting Transformer: Softmax Attention with a Forget Gatecited by 50
  9. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 39
  10. 2025MesaNet: Sequence Modeling by Locally Optimal Test-Time Trainingcited by 37
  11. 2024Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothingcited by 17
  12. 2021How Does Momentum Benefit Deep Neural Networks Architecture Design? A Few Case Studiescited by 10
20 more
  1. 2025HiCache: A Plug-in Scaled-Hermite Upgrade for Taylor-Style Cache-then-Forecast Diffusion Accelerationcited by 8
  2. 2021Slower is Better: Revisiting the Forgetting Mechanism in LSTM for Slower Information Decaycited by 7
  3. 2024OpenDPD: An Open-Source End-to-End Learning & Benchmarking Framework for Wideband Power Amplifier Modeling and Digital Pre-Distortioncited by 7
  4. 2024Rodimus*: Breaking the Accuracy-Efficiency Trade-Off with Efficient Attentionscited by 6
  5. 2021Recurrent Neural Networks for Learning Long-term Temporal Dependencies with Reanalysis of Time Scale Representationcited by 4
  6. 2021Warming up recurrent neural networks to maximise reachable multistability greatly improves learningcited by 3
  7. 2021A Brief Survey of Associations Between Meta-Learning and General AIcited by 2
  8. 2021Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directionscited by 2
  9. 2021Oscillatory Fourier Neural Network: A Compact and Efficient Architecture for Sequential Processingcited by 2
  10. 2021Gates Are Not What You Need in RNNscited by 1
  11. 2023Machine Learning based prediction of Glucose Levels in Type 1 Diabetes Patients with the use of Continuous Glucose Monitoring Datacited by 1
  12. 2025DeltaDPD: Exploiting Dynamic Temporal Sparsity in Recurrent Neural Networks for Energy-Efficient Wideband Digital Predistortioncited by 1
  13. 2025Don't Pay Attentioncited by 1
  14. 2025Time-Scale Coupling Between States and Parameters in Recurrent Neural Networkscited by 1
  15. 2025JaneEye: A 12-nm 2K-FPS 18.9-μJ/Frame Event-based Eye Tracking Acceleratorcited by 1
  16. 2026Learning to Forget: Continual Learning with Adaptive Weight Decaycited by 1
  17. 2022Learning Human Cognitive Appraisal Through Reinforcement Memory Unitcited by 0
  18. 2022Fast Saturating Gate for Learning Long Time Scales with Recurrent Neural Networkscited by 0
  19. 2022Embedding Compression for Text Classification Using Dictionary Screeningcited by 0
  20. 2026On the Importance of Multistability for Horizon Generalization in Reinforcement Learningcited by 0