mlsanityData to 7 Oct 2026

Bridging the Empirical-Theoretical Gap in Neural Network Formal Language Learning Using Minimum Description Length

arXiv 2402.10013 ↗First posted 15 Feb 2024cs.CL cs.FLcited by 14

Not on the map. Why?

Abstract (arXiv)

Neural networks offer good approximation to many tasks but consistently fail to reach perfect generalization, even when theoretical work shows that such perfect solutions can be expressed by certain architectures. Using the task of formal language learning, we focus on one simple formal language and show that the theoretically correct solution is in fact not an optimum of commonly used objectives -- even with regularization techniques that according to common wisdom should lead to simple weights and good generalization (L1, L2) or other meta-heuristics (early-stopping, dropout). On the other hand, replacing standard targets with the Minimum Description Length objective (MDL) results in the correct solution being an optimum.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 11 of its 39 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

11 of 39 references.

References · 11

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  3. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  4. 2018On the Practical Computational Power of Finite Precision RNNs for Language Recognitioncited by 117
  5. 2017Bayesian Compression for Deep Learningcited by 116
  6. 2015Inferring Algorithmic Patterns with Stack-Augmented Recurrent Netscited by 76
  7. 2020A Formal Hierarchy of RNN Architecturescited by 61
  8. 2019LSTM Networks Can Perform Dynamic Countingcited by 45
  9. 2020On the Linguistic Capacity of Real-Time Counter Automatacited by 16
  10. 2022Exploring the Long-Term Generalization of Counting Behavior in RNNscited by 7
  11. 2023Benchmarking Neural Network Generalization for Grammar Inductioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 6.

14 papers.

Sort:

  1. 2025How Linguistics Learned to Stop Worrying and Love the Language Modelscited by 50
  2. 2025Large Language Models as Proxies for Theories of Human Linguistic Cognitioncited by 10
  3. 2025Meta-Learning Neural Mechanisms rather than Bayesian Priorscited by 8
  4. 2025A Minimum Description Length Approach to Regularization in Neural Networkscited by 2
  5. 2025Language models as tools for investigating the distinction between possible and impossible natural languagescited by 0
  6. 2025Large language models have learned to use languagecited by 0
  7. 2025Large language models are not about natural languagecited by 0
  8. 2025Linguists should learn to love speech-based deep learning modelscited by 0
  9. 2026Are Language Models Models?cited by 0
  10. 2026Across the Levels of Analysis: Explaining Predictive Processing in Humans Requires More Than Machine-Estimated Probabilitiescited by 0
  11. 2026You Can't Fight in Here! This is BBS!cited by 0
  12. 2026Causal Drawbridges: Characterizing Gradient Blocking of Syntactic Islands in Transformer LMscited by 0
2 more
  1. 2026Not-So-Strange Love: Language Models and Generative Linguistic Theories are More Compatible than They Appearcited by 0
  2. 2026Comment on "Modeling rapid language learning by distilling Bayesian priors into artificial neural networks"cited by 0