Data to 7 Oct 2026

LSTM: A Search Space Odyssey

arXiv 1503.04069 ↗First posted 13 Mar 2015cs.NE cs.LGcited by 315

Not on the map. Why?

Abstract (arXiv)

Several variants of the Long Short-Term Memory (LSTM) architecture for recurrent neural networks have been proposed since its inception in 1995. In recent years, these networks have become the state-of-the-art models for a variety of machine learning problems. This has led to a renewed interest in understanding the role and utility of various computational components of typical LSTM variants. In this paper, we present the first large-scale analysis of eight LSTM variants on three representative tasks: speech recognition, handwriting recognition, and polyphonic music modeling. The hyperparameters of all LSTM variants for each task were optimized separately using random search, and their importance was assessed using the powerful fANOVA framework. In total, we summarize the results of 5400 experimental runs (≈ 15 years of CPU time), which makes our study the largest of its kind on LSTM networks. Our results show that none of the variants can improve upon the standard LSTM architecture significantly, and demonstrate the forget gate and the output activation function to be its most critical components. We further observe that the studied hyperparameters are virtually independent and derive guidelines for their efficient adjustment.

Key facts. Cited by 315 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Mar 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 82; 2022: 63; 2023: 51; 2024: 45; 2025: 50; 2026: 24.

315 papers.

Sort:

By area:

2021Deep Neural Networks and Tabular Data: A Surveycited by 489
2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
2024Titans: Learning to Memorize at Test Timecited by 303
2021Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Timecited by 140
2022Liquid Structural State-Space Modelscited by 134
2021A Survey of Visual Transformerscited by 117
2023AR-Diffusion: Auto-Regressive Diffusion Model for Text Generationcited by 113
2022Block-Recurrent Transformerscited by 107
2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 100
2023Generative Time Series Forecasting with Diffusion, Denoise, and Disentanglementcited by 89
2022Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoisecited by 79

38 more

2023Machine learning with data assimilation and uncertainty quantification for dynamical systems: a reviewcited by 74
2021Z-GCNETs: Time Zigzags at Graph Convolutional Networks for Time Series Forecastingcited by 67
2021Graph-Based Deep Learning for Medical Diagnosis and Analysis: Past, Present and Futurecited by 67
2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 63
2022Cross-modal Contrastive Learning for Speech Translationcited by 61
2022Do Current Multi-Task Optimization Methods in Deep Learning Even Help?cited by 60
2025It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimizationcited by 60
2024Language Repository for Long Video Understandingcited by 55
2023A Survey on Time-Series Pre-Trained Modelscited by 52
2024Toward Visual Grounding: A Surveycited by 52
2025Forgetting Transformer: Softmax Attention with a Forget Gatecited by 50
2022Rashomon Capacity: A Metric for Predictive Multiplicity in Classificationcited by 46
2022Deep Learning Approaches on Image Captioning: A Reviewcited by 43
2021Federated Learning Meets Natural Language Processing: A Surveycited by 42
2023Maintaining Plasticity in Deep Continual Learningcited by 42
2021Unsupervised Deep Anomaly Detection for Multi-Sensor Time-Series Signalscited by 41
2021Mosaic Flows: A Transferable Deep Learning Framework for Solving PDEs on Unseen Domainscited by 37
2022STCrowd: A Multimodal Dataset for Pedestrian Perception in Crowded Scenescited by 35
2022ScanEnts3D: Exploiting Phrase-to-3D-Object Correspondences for Improved Visio-Linguistic Models in 3D Scenescited by 32
2022Green Learning: Introduction, Examples and Outlookcited by 30
2021Self-supervised Autoregressive Domain Adaptation for Time Series Datacited by 29
2021Dual-side Sparse Tensor Corecited by 27
2021A comparison of LSTM and GRU networks for learning symbolic sequencescited by 24
2021A Survey on Machine Learning Techniques for Source Code Analysiscited by 23
2021Google Neural Network Models for Edge Devices: Analyzing and Mitigating Machine Learning Inference Bottleneckscited by 22
2023Efficient Robustness Assessment via Adversarial Spatial-Temporal Focus on Videoscited by 20
2023Scene Synthesis from Human Motioncited by 20
2021Augmented Shortcuts for Vision Transformerscited by 19
2021Interpreting Deep Learning Models in Natural Language Processing: A Reviewcited by 19
2021Recurrent Off-policy Baselines for Memory-based Continuous Controlcited by 19
2023Exploring the Promise and Limits of Real-Time Recurrent Learningcited by 19
2024xLSTM-Mixer: Multivariate Time Series Forecasting by Mixing via Scalar Memoriescited by 18
2024Prompt Public Large Language Models to Synthesize Data for Private On-device Applicationscited by 17
2024Best Practices for a Handwritten Text Recognition Systemcited by 17
2021Recurrent Neural Network from Adder's Perspective: Carry-lookahead RNNcited by 16
2023Deep Learning for Steganalysis of Diverse Data Types: A review of methods, taxonomy, challenges and future directionscited by 16
2021HeterPS: Distributed Deep Learning With Reinforcement Learning Based Scheduling in Heterogeneous Environmentscited by 15
2022FECAM: Frequency Enhanced Channel Attention Mechanism for Time Series Forecastingcited by 15

50 most cited of 315 All 315 (JSON)