# Biasless Language Models Learn Unnaturally: How LLMs Fail to Distinguish the Possible from the Impossible

arXiv: https://arxiv.org/abs/2510.07178 · HTML: https://mlsanity.com/paper/2510.07178/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 51 references that are in the map.
- First posted on arXiv 8 Oct 2025.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Colorless green recurrent networks dream hierarchically](https://mlsanity.com/paper/1803.11138/) (2018)
- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://mlsanity.com/paper/2104.06644/) (2021)
- [Syntactic Structure from Deep Learning](https://mlsanity.com/paper/2004.10827/) (2020)
- [Why do language models perform worse for morphologically complex languages?](https://mlsanity.com/paper/2411.14198/) (2024)
- [How Linguistics Learned to Stop Worrying and Love the Language Models](https://mlsanity.com/paper/2501.17047/) (2025)
- [On the proper role of linguistically-oriented deep net analysis in linguistic theorizing](https://mlsanity.com/paper/2106.08694/) (2021)
- [Word Order Does Matter (And Shuffled Language Models Know It)](https://mlsanity.com/paper/2203.10995/) (2022)
- [Can Language Models Learn Typologically Implausible Languages?](https://mlsanity.com/paper/2502.12317/) (2025)
- [Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMs](https://mlsanity.com/paper/2502.18795/) (2025)
- [A Cross-Linguistic Pressure for Uniform Information Density in Word Order](https://mlsanity.com/paper/2306.03734/) (2023)
- [DictaBERT: A State-of-the-Art BERT Suite for Modern Hebrew](https://mlsanity.com/paper/2308.16687/) (2023)
- [Kallini et al. (2024) do not compare impossible languages with constituency-based ones](https://mlsanity.com/paper/2410.12271/) (2024)
- [Large Language Models as Proxies for Theories of Human Linguistic Cognition](https://mlsanity.com/paper/2502.07687/) (2025)
- [Linguistic Structure from a Bottleneck on Sequential Information Processing](https://mlsanity.com/paper/2405.12109/) (2024)
- [Large Language Model probabilities cannot distinguish between possible and impossible language](https://mlsanity.com/paper/2509.15114/) (2025)
- [No Such Thing as a General Learner: Language models and their dual optimization](https://mlsanity.com/paper/2408.09544/) (2024)

## Built on by (most cited first, 3 of 3)

- [Function Words as Statistical Cues for Language Learning](https://mlsanity.com/paper/2601.21191/) (2026)
- [Language Re-generation: An investigation into information locality effects on reconstruction](https://mlsanity.com/paper/2607.10268/) (2026)
- [Informational Antilocality and the Locality Bias in LLMs](https://mlsanity.com/paper/2608.27760/) (2026)

## Abstract (arXiv)

Are large language models (LLMs) sensitive to the distinction between humanly possible and impossible languages? This question was recently used in a broader debate on whether LLMs and humans share the same innate learning biases. Previous work has answered it in the positive by comparing LLM learning curves on existing language datasets and on "impossible" datasets derived from them via various perturbation functions. Using the same methodology, we examine this claim on a wider set of languages and impossible perturbations. We find that in most cases, GPT-2 learns each language and its impossible counterpart equally easily, in contrast to previous findings. We also apply a more lenient condition by testing whether GPT-2 provides any kind of separation between the whole sets of natural vs. impossible languages, based on cross-linguistic variance in metrics derived from the learning curves. Taken together, these perspectives show that GPT-2 provides no systematic separation between the possible and the impossible.