mlsanityData to 14 Oct 2026

When transformers learn "impossible" languages, what do they learn?

arXiv 2606.30815 ↗First posted 29 Jun 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent work suggests that transformer language models show a bias towards human languages over unnatural ("impossible") languages argued to be unacquirable by humans. However, this literature has largely based these claims on differences in sample efficiency and test-set perplexity, rather than on direct evaluations of the linguistic capacities that could plausibly explain non-attestation in human languages. We evaluate two theoretically motivated linking hypotheses: impossibility arising from deficiencies in grammatical sensitivity or generative production. Using GPT-2 style models trained on perturbed "impossible" variants of English, we measure sensitivity to grammaticality using BLiMP minimal pairs, finding that model performance exhibits only gradual degradation, mediated by the language's information locality. In contrast, these models exhibited pronounced failures in generation, producing substantially fewer high-quality sentences at longer lengths. Together, these results suggest generative deficiency and transmission failures as a plausible linking hypothesis between language model behaviour and non-attestation of impossible languages.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 34 references that are in the map. First posted on arXiv 29 Jun 2026.

Builds on

15 of 34 references.

References · 15

  1. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  2. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  3. 2024A Note on Shumailov et al. (2024): `AI Models Collapse When Trained on Recursively Generated Data'cited by 115
  4. 2023The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Textcited by 100
  5. 2020Syntactic Structure from Deep Learningcited by 99
  6. 2024Understanding Emergent Abilities of Language Models from the Loss Perspectivecited by 94
  7. 2025How Linguistics Learned to Stop Worrying and Love the Language Modelscited by 51
  8. 2024Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 48
  9. 2023Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokenscited by 41
  10. 2025Can Language Models Learn Typologically Implausible Languages?cited by 23
  11. 2025Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMscited by 20
  12. 2022Evaluating Distributional Distortion in Neural Language Modelingcited by 19
3 more
  1. 2025What Can String Probability Tell Us About Grammaticality?cited by 16
  2. 2025Information Locality as an Inductive Bias for Neural Language Modelscited by 14
  3. 2026Form and Meaning in Intrinsic Multilingual Evaluationscited by 2

Built on by

Not cited yet.