# A comparison of LSTM and GRU networks for learning symbolic sequences

arXiv: https://arxiv.org/abs/2107.02248 · HTML: https://mlsanity.com/paper/2107.02248/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 36 references that are in the map.
- First posted on arXiv 5 Jul 2021.

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [N-BEATS: Neural basis expansion analysis for interpretable time series forecasting](https://mlsanity.com/paper/1905.10437/) (2019)
- [LSTM: A Search Space Odyssey](https://mlsanity.com/paper/1503.04069/) (2015)
- [Deep Learning for Symbolic Mathematics](https://mlsanity.com/paper/1912.01412/) (2019)
- [Sequence to Sequence -- Video to Text](https://mlsanity.com/paper/1505.00487/) (2015)
- [Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)](https://mlsanity.com/paper/1412.6632/) (2014)
- [Principles and Algorithms for Forecasting Groups of Time Series: Locality and Globality](https://mlsanity.com/paper/2008.00444/) (2020)
- [TimeNet: Pre-trained deep recurrent neural network for time series classification](https://mlsanity.com/paper/1706.08838/) (2017)
- [Time Series Forecasting Using LSTM Networks: A Symbolic Approach](https://mlsanity.com/paper/2003.05672/) (2020)
- [The Effectiveness of Discretization in Forecasting: An Empirical Study on Neural Time Series Models](https://mlsanity.com/paper/2005.10111/) (2020)
- [Learning to Discover Efficient Mathematical Identities](https://mlsanity.com/paper/1406.1584/) (2014)
- [Neural Sequence-to-grid Module for Learning Symbolic Rules](https://mlsanity.com/paper/2101.04921/) (2021)

## Built on by (most cited first, 24 of 24)

- [A Federated Learning-based Industrial Health Prognostics for Heterogeneous Edge Devices using Matched Feature Extraction](https://mlsanity.com/paper/2305.07854/) (2023)
- [Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Review](https://mlsanity.com/paper/2311.11883/) (2023)
- [Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mamba](https://mlsanity.com/paper/2503.18970/) (2025)
- [Comprehensive Attribute Encoding and Dynamic LSTM HyperModels for Outcome Oriented Predictive Business Process Monitoring](https://mlsanity.com/paper/2506.03696/) (2025)
- [Cerberus: Exploring Federated Prediction of Security Events](https://mlsanity.com/paper/2209.03050/) (2022)
- [A Comparison of Deep Learning Architectures for Spacecraft Anomaly Detection](https://mlsanity.com/paper/2403.12864/) (2024)
- [ChronoGAN: Supervised and Embedded Generative Adversarial Networks for Time Series Generation](https://mlsanity.com/paper/2409.14013/) (2024)
- [Mathematical Formalism for Memory Compression in Selective State Space Models](https://mlsanity.com/paper/2410.03158/) (2024)
- [Modeling and Predicting Transistor Aging under Workload Dependency using Machine Learning](https://mlsanity.com/paper/2207.04134/) (2022)
- [RIOT: Recursive Inertial Odometry Transformer for Localisation from Low-Cost IMU Measurements](https://mlsanity.com/paper/2303.01641/) (2023)
- [A generalised multi-factor deep learning electricity load forecasting model for wildfire-prone areas](https://mlsanity.com/paper/2304.10686/) (2023)
- [Mobile Traffic Prediction at the Edge Through Distributed and Deep Transfer Learning](https://mlsanity.com/paper/2310.14456/) (2023)
- [GNN-based Decentralized Perception in Multirobot Systems for Predicting Worker Actions](https://mlsanity.com/paper/2501.04193/) (2025)
- [Minion Gated Recurrent Unit for Continual Learning](https://mlsanity.com/paper/2503.06175/) (2025)
- [Probing Quantum Spin Systems with Kolmogorov-Arnold Neural Network Quantum States](https://mlsanity.com/paper/2506.01891/) (2025)
- [Robustness of Spatio-temporal Graph Neural Networks for Fault Location in Partially Observable Distribution Grids](https://mlsanity.com/paper/2604.20403/) (2026)
- [Using LSTM and GRU With a New Dataset for Named Entity Recognition in the Arabic Language](https://mlsanity.com/paper/2304.03399/) (2023)
- [Deep-seeded Clustering for Emotion Recognition from Wearable Physiological Sensors](https://mlsanity.com/paper/2308.09013/) (2023)
- [Enhancing Microgrid Performance Prediction with Attention-based Deep Learning Models](https://mlsanity.com/paper/2407.14984/) (2024)
- [Are ECGs enough? Deep learning classification of pulmonary embolism using electrocardiograms](https://mlsanity.com/paper/2503.08960/) (2025)
- [Enhancing Tropical Cyclone Path Forecasting with an Improved Transformer Network](https://mlsanity.com/paper/2505.00495/) (2025)
- [Generalization of Graph Neural Network Models for Distribution Grid Fault Detection](https://mlsanity.com/paper/2510.03571/) (2025)
- [SymSeqBench: a unified framework for the generation and analysis of rule-based symbolic sequences and datasets](https://mlsanity.com/paper/2512.24977/) (2025)
- [ROTE: Benchmarking Neural Memorization on Complexity-Controlled Symbolic Sequences](https://mlsanity.com/paper/2609.31918/) (2026)

## Abstract (arXiv)

We explore the architecture of recurrent neural networks (RNNs) by studying the complexity of string sequences it is able to memorize. Symbolic sequences of different complexity are generated to simulate RNN training and study parameter configurations with a view to the network's capability of learning and inference. We compare Long Short-Term Memory (LSTM) networks and gated recurrent units (GRUs). We find that an increase in RNN depth does not necessarily result in better memorization capability when the training time is constrained. Our results also indicate that the learning rate and the number of units per layer are among the most important hyper-parameters to be tuned. Generally, GRUs outperform LSTM networks on low-complexity sequences while on high-complexity sequences LSTMs perform better.