mlsanityData to 14 Oct 2026

Minimum Description Length Recurrent Neural Networks

arXiv 2111.00600 ↗First posted 31 Oct 2021cs.CLcited by 24

Not on the map. Why?

Abstract (arXiv)

We train neural networks to optimize a Minimum Description Length score, i.e., to balance between the complexity of the network and its accuracy at a task. We show that networks optimizing this objective function master tasks involving memory challenges and go beyond context-free languages. These learners master languages such as anbn, anbncn, anb2n, anbmcn+m, and they perform addition. Moreover, they often do so with 100% accuracy. The networks are small, and their inner workings are transparent. We thus provide formal proofs that their perfect accuracy holds not only on a given test set, but for any input sequence. To our knowledge, no other connectionist model has been shown to capture the underlying grammars for these languages in full generality.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 12 of its 50 references that are in the map. First posted on arXiv 31 Oct 2021.

Builds on

12 of 50 references.

References · 12

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  3. 2014Deep Learning in Neural Networks: An Overviewcited by 770
  4. 2014Neural Turing Machinescited by 644
  5. 2018On the Practical Computational Power of Finite Precision RNNs for Language Recognitioncited by 117
  6. 2019The emergence of number and syntax units in LSTM language modelscited by 94
  7. 2015Inferring Algorithmic Patterns with Stack-Augmented Recurrent Netscited by 76
  8. 2019Weight Agnostic Neural Networkscited by 71
  9. 2017Extracting Automata from Recurrent Neural Networks Using Queries and Counterexamplescited by 66
  10. 2019Memory-Augmented Recurrent Neural Networks Can Learn Generalized Dyck Languagescited by 31
  11. 2017One Model for the Learning of Languagecited by 22
  12. 2017An Empirical Evaluation of Rule Extraction from Recurrent Neural Networkscited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 0; 2023: 3; 2024: 6; 2025: 8; 2026: 6.

24 papers.

Sort:

  1. 2025How Linguistics Learned to Stop Worrying and Love the Language Modelscited by 51
  2. 2025Large Language Models as Proxies for Theories of Human Linguistic Cognitioncited by 10
  3. 2024Towards Interpretable Physical-Conceptual Catchment-Scale Hydrological Modeling using the Mass-Conserving-Perceptroncited by 5
  4. 2024Using Machine Learning to Discover Parsimonious and Physically-Interpretable Representations of Catchment-Scale Rainfall-Runoff Dynamicscited by 4
  5. 2023Benchmarking Neural Network Generalization for Grammar Inductioncited by 3
  6. 2024Neural Networks Generalize on Low Complexity Datacited by 3
  7. 2021How Do Neural Sequence Models Generalize? Local and Global Context Cues for Out-of-Distribution Predictioncited by 2
  8. 2025A Minimum Description Length Approach to Regularization in Neural Networkscited by 2
  9. 2023The Quo Vadis of the Relationship between Language and Large Language Modelscited by 1
  10. 2023Minimum Description Length Hopfield Networkscited by 1
  11. 2025Chaotic Map based Compression Approach to Classificationcited by 1
  12. 2026Grokking From Abstraction to Intelligencecited by 1
12 more
  1. 2024Precision, Stability, and Generalization: A Comprehensive Assessment of RNNs learnability capability for Classifying Counter and Dyck Languagescited by 0
  2. 2024Exploring Learnability in Memory-Augmented Recurrent Neural Networks: Precision, Stability, and Empirical Insightscited by 0
  3. 2024On the Compatibility of Generative AI and Generative Linguisticscited by 0
  4. 2025Language models as tools for investigating the distinction between possible and impossible natural languagescited by 0
  5. 2025Large language models have learned to use languagecited by 0
  6. 2025Large language models are not about natural languagecited by 0
  7. 2025Linguists should learn to love speech-based deep learning modelscited by 0
  8. 2026Are Language Models Models?cited by 0
  9. 2026Across the Levels of Analysis: Explaining Predictive Processing in Humans Requires More Than Machine-Estimated Probabilitiescited by 0
  10. 2026You Can't Fight in Here! This is BBS!cited by 0
  11. 2026Not-So-Strange Love: Language Models and Generative Linguistic Theories are More Compatible than They Appearcited by 0
  12. 2026A Gated Catchment Hydrology Model (GCHM) for Physically-Interpretable Rainfall-Evapotranspiration-Runoff Simulation Across Diverse Hydroclimatic Regimescited by 0