# Kallini et al. (2024) do not compare impossible languages with constituency-based ones

arXiv: https://arxiv.org/abs/2410.12271 · HTML: https://mlsanity.com/paper/2410.12271/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 26 references that are in the map.
- First posted on arXiv 16 Oct 2024.

## Builds on (most cited first, 5 of 5)

- [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://mlsanity.com/paper/1611.01368/) (2016)
- [Does syntax need to grow on trees? Sources of hierarchical inductive bias in sequence-to-sequence networks](https://mlsanity.com/paper/2001.03632/) (2020)
- [Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humans](https://mlsanity.com/paper/2006.11098/) (2020)
- [Can neural networks acquire a structural bias from raw linguistic data?](https://mlsanity.com/paper/2007.06761/) (2020)
- [Mission: Impossible Language Models](https://mlsanity.com/paper/2401.06416/) (2024)

## Built on by (most cited first, 15 of 15)

- [How Linguistics Learned to Stop Worrying and Love the Language Models](https://mlsanity.com/paper/2501.17047/) (2025)
- [Can Language Models Learn Typologically Implausible Languages?](https://mlsanity.com/paper/2502.12317/) (2025)
- [Large Language Models as Proxies for Theories of Human Linguistic Cognition](https://mlsanity.com/paper/2502.07687/) (2025)
- [Large Language Model probabilities cannot distinguish between possible and impossible language](https://mlsanity.com/paper/2509.15114/) (2025)
- [Which Word Orders Facilitate Length Generalization in LMs? An Investigation with GCG-Based Artificial Languages](https://mlsanity.com/paper/2510.12722/) (2025)
- [Biasless Language Models Learn Unnaturally: How LLMs Fail to Distinguish the Possible from the Impossible](https://mlsanity.com/paper/2510.07178/) (2025)
- [Language models as tools for investigating the distinction between possible and impossible natural languages](https://mlsanity.com/paper/2512.09394/) (2025)
- [Large language models have learned to use language](https://mlsanity.com/paper/2512.12447/) (2025)
- [Large language models are not about natural language](https://mlsanity.com/paper/2512.13441/) (2025)
- [Linguists should learn to love speech-based deep learning models](https://mlsanity.com/paper/2512.14506/) (2025)
- [Are Language Models Models?](https://mlsanity.com/paper/2601.10421/) (2026)
- [Across the Levels of Analysis: Explaining Predictive Processing in Humans Requires More Than Machine-Estimated Probabilities](https://mlsanity.com/paper/2604.09466/) (2026)
- [You Can't Fight in Here! This is BBS!](https://mlsanity.com/paper/2604.09501/) (2026)
- [Not-So-Strange Love: Language Models and Generative Linguistic Theories are More Compatible than They Appear](https://mlsanity.com/paper/2605.10061/) (2026)
- [Typological Alignment of Stack-Based Language Models on Mildly Context-Sensitive Artificial Languages](https://mlsanity.com/paper/2610.02040/) (2026)

## Abstract (arXiv)

A central goal of linguistic theory is to find a precise characterization of the notion "possible human language", in the form of a computational device that is capable of describing all and only the languages that can be acquired by a typically developing human child. The success of recent large language models (LLMs) in NLP applications arguably raises the possibility that LLMs might be computational devices that meet this goal. This would only be the case if, in addition to succeeding in learning human languages, LLMs struggle to learn "impossible" human languages. Kallini et al. (2024; "Mission: Impossible Language Models", Proc. ACL) conducted experiments aiming to test this by training GPT-2 on a variety of synthetic languages, and found that it learns some more successfully than others. They present these asymmetries as support for the idea that LLMs' inductive biases align with what is regarded as "possible" for human languages, but the most significant comparison has a confound that makes this conclusion unwarranted. In this paper I explain the confound and suggest some ways forward towards constructing a comparison that appropriately tests the underlying issue.