# On Evaluating the Generalization of LSTM Models in Formal Languages

arXiv: https://arxiv.org/abs/1811.01001 · HTML: https://mlsanity.com/paper/1811.01001/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Nov 2018.

## Built on by (most cited first, 17 of 17)

- [Evaluating the World Model Implicit in a Generative Model](https://mlsanity.com/paper/2406.03689/) (2024)
- [Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functions](https://mlsanity.com/paper/2211.12316/) (2022)
- [What Has a Foundation Model Found? Using Inductive Bias to Probe for World Models](https://mlsanity.com/paper/2507.06952/) (2025)
- [The Expressive Capacity of State Space Models: A Formal Language Perspective](https://mlsanity.com/paper/2405.17394/) (2024)
- [Training Neural Networks as Recognizers of Formal Languages](https://mlsanity.com/paper/2411.07107/) (2024)
- [What Languages are Easy to Language-Model? A Perspective from Learning Probabilistic Regular Languages](https://mlsanity.com/paper/2406.04289/) (2024)
- [Can RNNs learn Recursive Nested Subject-Verb Agreements?](https://mlsanity.com/paper/2101.02258/) (2021)
- [Improving Intrinsic Exploration by Creating Stationary Objectives](https://mlsanity.com/paper/2310.18144/) (2023)
- [Introducing Rhetorical Parallelism Detection: A New Task with Datasets, Metrics, and Baselines](https://mlsanity.com/paper/2312.00100/) (2023)
- [Understanding Robust Generalization in Learning Regular Languages](https://mlsanity.com/paper/2202.09717/) (2022)
- [What do Large Language Models Learn beyond Language?](https://mlsanity.com/paper/2210.12302/) (2022)
- [Are Transformers a Modern Version of ELIZA? Observations on French Object Verb Agreement](https://mlsanity.com/paper/2109.10133/) (2021)
- [Evaluating Transformer's Ability to Learn Mildly Context-Sensitive Languages](https://mlsanity.com/paper/2309.00857/) (2023)
- [Multi-Objective Evolutionary Neural Architecture Search for Recurrent Neural Networks](https://mlsanity.com/paper/2403.11173/) (2024)
- [Extracting Moore Machines from Transformers using Queries and Counterexamples](https://mlsanity.com/paper/2410.06045/) (2024)
- [Precision, Stability, and Generalization: A Comprehensive Assessment of RNNs learnability capability for Classifying Counter and Dyck Languages](https://mlsanity.com/paper/2410.03118/) (2024)
- [The Domain Mixed Unit: A New Neural Arithmetic Layer](https://mlsanity.com/paper/2509.08180/) (2025)

## Abstract (arXiv)

Recurrent Neural Networks (RNNs) are theoretically Turing-complete and established themselves as a dominant model for language processing. Yet, there still remains an uncertainty regarding their language learning capabilities. In this paper, we empirically evaluate the inductive learning capabilities of Long Short-Term Memory networks, a popular extension of simple RNNs, to learn simple formal languages, in particular anbn, anbncn, and anbncndn. We investigate the influence of various aspects of learning, such as training data regimes and model capacity, on the generalization to unobserved samples. We find striking differences in model performances under different training settings and highlight the need for careful analysis and assessment when making claims about the learning capabilities of neural network models.