# LLMs are not stochastic parrots: Evidence for meaning-mediated abstraction from conlang-like tasks

arXiv: https://arxiv.org/abs/2609.34187 · HTML: https://mlsanity.com/paper/2609.34187/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 81 references that are in the map.
- First posted on arXiv 28 Sep 2026.

## Builds on (most cited first, 25 of 25)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models](https://mlsanity.com/paper/2410.05229/) (2024)
- [2 OLMo 2 Furious](https://mlsanity.com/paper/2501.00656/) (2024)
- [Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve](https://mlsanity.com/paper/2309.13638/) (2023)
- [Hierarchical Reasoning Model](https://mlsanity.com/paper/2506.21734/) (2025)
- [Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens](https://mlsanity.com/paper/2401.17377/) (2024)
- [Does Prompt Formatting Have Any Impact on LLM Performance?](https://mlsanity.com/paper/2411.10541/) (2024)
- [Optimizing Length Compression in Large Reasoning Models](https://mlsanity.com/paper/2506.14755/) (2025)
- [Energy-Based Transformers are Scalable Learners and Thinkers](https://mlsanity.com/paper/2507.02092/) (2025)
- [Dissociating language and thought in large language models](https://mlsanity.com/paper/2301.06627/) (2023)
- [TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning](https://mlsanity.com/paper/2509.25760/) (2025)
- [An enriched category theory of language: from syntax to semantics](https://mlsanity.com/paper/2106.07890/) (2021)
- [Prompt Chaining or Stepwise Prompt? Refinement in Text Summarization](https://mlsanity.com/paper/2406.00507/) (2024)
- [Instruction Following without Instruction Tuning](https://mlsanity.com/paper/2409.14254/) (2024)
- [Neologism Learning for Controllability and Self-Verbalization](https://mlsanity.com/paper/2510.08506/) (2025)
- [Beyond the Black Box: A Survey on the Theory and Mechanism of Large Language Models](https://mlsanity.com/paper/2601.02907/) (2026)
- ["Genlangs" and Zipf's Law: Do languages generated by ChatGPT statistically look human?](https://mlsanity.com/paper/2304.12191/) (2023)
- [Tokens, the oft-overlooked appetizer: Large language models, the distributional hypothesis, and meaning](https://mlsanity.com/paper/2412.10924/) (2024)
- [Lexinvariant Language Models](https://mlsanity.com/paper/2305.16349/) (2023)
- [Billions of Sketches Reveal Hidden Cultural Variation in Human Concepts](https://mlsanity.com/paper/2607.07267/) (2026)

## Built on by (most cited first, 1 of 1)

- [Contextual trajectory and incremental contextual displacement: Towards using LLMs to understand dynamic, utterance-specific meaning construction](https://mlsanity.com/paper/2610.00840/) (2026)

## Abstract (arXiv)

The strong version of the stochastic parrot argument claims that, although large language models (LLMs) may exceed rote regurgitation, they cannot move beyond statistical pattern matching into abstraction or reasoning, remaining ontologically near the lower bound of pattern reuse despite producing alluringly fluent text. We test this hypothesis using conlang-like tasks. Several LLMs are given only natural-language descriptions of fictional languages that subvert prominent superficial patterns in training data by combining statistically uncommon and unattested features. Crucially, no example outputs are given. We argue that if the models exhibit rule-following behaviour, they cannot be relying solely on superficial statistical patterns; such patterns often work against the correct output. Instead, successful performance requires representations of the constraints specified in the prompt. Across three complementary task families, models systematically move in the meaning-predicted direction: they distinguish prompt exposure from instructed use, alter semantic relationships in response to novel constraints, and sometimes produce exact matches to complex translation answer keys. Although performance varies across the spectrum of models used, these results provide evidence for meaning-mediated abstraction in LLMs and refute the strong stochastic parrot hypothesis. Our work shows that, under appropriate architectural and contextual constraints, statistical learning can produce meaning-mediated abstractions, although generation remains strongly constrained by superficial plausibility. We discuss implications for model development and for understanding how increasingly abstract representations may emerge from plausible-text-generation objectives.