mlsanityData to 14 Oct 2026

Kallini et al. (2024) do not compare impossible languages with constituency-based ones

arXiv 2410.12271 ↗First posted 16 Oct 2024cs.CL cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

A central goal of linguistic theory is to find a precise characterization of the notion "possible human language", in the form of a computational device that is capable of describing all and only the languages that can be acquired by a typically developing human child. The success of recent large language models (LLMs) in NLP applications arguably raises the possibility that LLMs might be computational devices that meet this goal. This would only be the case if, in addition to succeeding in learning human languages, LLMs struggle to learn "impossible" human languages. Kallini et al. (2024; "Mission: Impossible Language Models", Proc. ACL) conducted experiments aiming to test this by training GPT-2 on a variety of synthetic languages, and found that it learns some more successfully than others. They present these asymmetries as support for the idea that LLMs' inductive biases align with what is regarded as "possible" for human languages, but the most significant comparison has a confound that makes this conclusion unwarranted. In this paper I explain the confound and suggest some ways forward towards constructing a comparison that appropriately tests the underlying issue.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 5 of its 26 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

5 of 26 references.

References · 5

  1. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 354
  2. 2020Does syntax need to grow on trees? Sources of hierarchical inductive bias in sequence-to-sequence networkscited by 72
  3. 2020Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humanscited by 42
  4. 2020Can neural networks acquire a structural bias from raw linguistic data?cited by 35
  5. 2024Mission: Impossible Language Modelscited by 32

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 5.

15 papers.

Sort:

  1. 2025How Linguistics Learned to Stop Worrying and Love the Language Modelscited by 51
  2. 2025Can Language Models Learn Typologically Implausible Languages?cited by 23
  3. 2025Large Language Models as Proxies for Theories of Human Linguistic Cognitioncited by 10
  4. 2025Large Language Model probabilities cannot distinguish between possible and impossible languagecited by 5
  5. 2025Which Word Orders Facilitate Length Generalization in LMs? An Investigation with GCG-Based Artificial Languagescited by 4
  6. 2025Biasless Language Models Learn Unnaturally: How LLMs Fail to Distinguish the Possible from the Impossiblecited by 3
  7. 2025Language models as tools for investigating the distinction between possible and impossible natural languagescited by 0
  8. 2025Large language models have learned to use languagecited by 0
  9. 2025Large language models are not about natural languagecited by 0
  10. 2025Linguists should learn to love speech-based deep learning modelscited by 0
  11. 2026Are Language Models Models?cited by 0
  12. 2026Across the Levels of Analysis: Explaining Predictive Processing in Humans Requires More Than Machine-Estimated Probabilitiescited by 0
3 more
  1. 2026You Can't Fight in Here! This is BBS!cited by 0
  2. 2026Not-So-Strange Love: Language Models and Generative Linguistic Theories are More Compatible than They Appearcited by 0
  3. 2026Typological Alignment of Stack-Based Language Models on Mildly Context-Sensitive Artificial Languagescited by 0