# Minimum Description Length Recurrent Neural Networks

arXiv: https://arxiv.org/abs/2111.00600 · HTML: https://mlsanity.com/paper/2111.00600/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 50 references that are in the map.
- First posted on arXiv 31 Oct 2021.

## Builds on (most cited first, 12 of 12)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [On the Practical Computational Power of Finite Precision RNNs for Language Recognition](https://mlsanity.com/paper/1805.04908/) (2018)
- [The emergence of number and syntax units in LSTM language models](https://mlsanity.com/paper/1903.07435/) (2019)
- [Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets](https://mlsanity.com/paper/1503.01007/) (2015)
- [Weight Agnostic Neural Networks](https://mlsanity.com/paper/1906.04358/) (2019)
- [Extracting Automata from Recurrent Neural Networks Using Queries and Counterexamples](https://mlsanity.com/paper/1711.09576/) (2017)
- [Memory-Augmented Recurrent Neural Networks Can Learn Generalized Dyck Languages](https://mlsanity.com/paper/1911.03329/) (2019)
- [One Model for the Learning of Language](https://mlsanity.com/paper/1711.06301/) (2017)
- [An Empirical Evaluation of Rule Extraction from Recurrent Neural Networks](https://mlsanity.com/paper/1709.10380/) (2017)

## Built on by (most cited first, 24 of 24)

- [How Linguistics Learned to Stop Worrying and Love the Language Models](https://mlsanity.com/paper/2501.17047/) (2025)
- [Large Language Models as Proxies for Theories of Human Linguistic Cognition](https://mlsanity.com/paper/2502.07687/) (2025)
- [Towards Interpretable Physical-Conceptual Catchment-Scale Hydrological Modeling using the Mass-Conserving-Perceptron](https://mlsanity.com/paper/2401.14521/) (2024)
- [Using Machine Learning to Discover Parsimonious and Physically-Interpretable Representations of Catchment-Scale Rainfall-Runoff Dynamics](https://mlsanity.com/paper/2412.04845/) (2024)
- [Benchmarking Neural Network Generalization for Grammar Induction](https://mlsanity.com/paper/2308.08253/) (2023)
- [Neural Networks Generalize on Low Complexity Data](https://mlsanity.com/paper/2409.12446/) (2024)
- [How Do Neural Sequence Models Generalize? Local and Global Context Cues for Out-of-Distribution Prediction](https://mlsanity.com/paper/2111.03108/) (2021)
- [A Minimum Description Length Approach to Regularization in Neural Networks](https://mlsanity.com/paper/2505.13398/) (2025)
- [The Quo Vadis of the Relationship between Language and Large Language Models](https://mlsanity.com/paper/2310.11146/) (2023)
- [Minimum Description Length Hopfield Networks](https://mlsanity.com/paper/2311.06518/) (2023)
- [Chaotic Map based Compression Approach to Classification](https://mlsanity.com/paper/2502.12302/) (2025)
- [Grokking From Abstraction to Intelligence](https://mlsanity.com/paper/2603.29262/) (2026)
- [Precision, Stability, and Generalization: A Comprehensive Assessment of RNNs learnability capability for Classifying Counter and Dyck Languages](https://mlsanity.com/paper/2410.03118/) (2024)
- [Exploring Learnability in Memory-Augmented Recurrent Neural Networks: Precision, Stability, and Empirical Insights](https://mlsanity.com/paper/2410.03154/) (2024)
- [On the Compatibility of Generative AI and Generative Linguistics](https://mlsanity.com/paper/2411.10533/) (2024)
- [Language models as tools for investigating the distinction between possible and impossible natural languages](https://mlsanity.com/paper/2512.09394/) (2025)
- [Large language models have learned to use language](https://mlsanity.com/paper/2512.12447/) (2025)
- [Large language models are not about natural language](https://mlsanity.com/paper/2512.13441/) (2025)
- [Linguists should learn to love speech-based deep learning models](https://mlsanity.com/paper/2512.14506/) (2025)
- [Are Language Models Models?](https://mlsanity.com/paper/2601.10421/) (2026)
- [Across the Levels of Analysis: Explaining Predictive Processing in Humans Requires More Than Machine-Estimated Probabilities](https://mlsanity.com/paper/2604.09466/) (2026)
- [You Can't Fight in Here! This is BBS!](https://mlsanity.com/paper/2604.09501/) (2026)
- [Not-So-Strange Love: Language Models and Generative Linguistic Theories are More Compatible than They Appear](https://mlsanity.com/paper/2605.10061/) (2026)
- [A Gated Catchment Hydrology Model (GCHM) for Physically-Interpretable Rainfall-Evapotranspiration-Runoff Simulation Across Diverse Hydroclimatic Regimes](https://mlsanity.com/paper/2610.14493/) (2026)

## Abstract (arXiv)

We train neural networks to optimize a Minimum Description Length score, i.e., to balance between the complexity of the network and its accuracy at a task. We show that networks optimizing this objective function master tasks involving memory challenges and go beyond context-free languages. These learners master languages such as anbn, anbncn, anb2n, anbmcn+m, and they perform addition. Moreover, they often do so with 100% accuracy. The networks are small, and their inner workings are transparent. We thus provide formal proofs that their perfect accuracy holds not only on a given test set, but for any input sequence. To our knowledge, no other connectionist model has been shown to capture the underlying grammars for these languages in full generality.