mlsanityData to 14 Oct 2026

Readability ne Learnability: Rethinking the Role of Simplicity in Training Small Language Models

arXiv 2510.13915 ↗First posted 15 Oct 2025cs.CL cs.AI cs.LGcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Recent studies suggest that very small language models (SLMs) can generate surprisingly coherent text when trained on simplified, child-directed corpora such as TinyStories. These findings have been interpreted as evidence that readability -- characterized by accessible vocabulary, familiar narrative structure, and simple syntax -- plays a key role in enabling such capabilities to emerge. In this paper, we challenge that interpretation. We construct synthetic datasets with matched structure but varied readability, and find that readability alone does not predict coherence or learning efficiency in SLMs. Models trained on complex, adult-level text perform comparably to those trained on simplified language, and even exhibit faster development of coherence during training. Instead, we show that statistical simplicity, as measured by n-gram diversity, is a stronger predictor of learnability. Our findings caution against the growing trend of anthropomorphizing language model training -- drawing parallels to human cognitive development without empirical basis -- and argue for more precise reasoning about what properties actually support capability emergence in small models.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 5 of its 26 references that are in the map. First posted on arXiv 15 Oct 2025.

Builds on

5 of 26 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Thinking beyond the anthropomorphic paradigm benefits LLM research13 Feb

Other references · 4

  1. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  2. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  3. 2022Talking About Large Language Modelscited by 175
  4. 2024Evaluating n-Gram Novelty of Language Models Using Rusty-DAWGcited by 25

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024From Babbling to Fluency: Evaluating the Evolution of Language Models in Terms of Human Language Acquisition
  2. 2025Rethinking the Role of Text Complexity in Language Model Pretraining
  3. 2025A Review of Developmental Interpretability in Large Language Models
  4. 2023A Latent Space Theory for Emergent Abilities in Large Language Models
  5. 2025Emergent Abilities in Large Language Models: A Survey
  6. 2024Observational Scaling Laws and the Predictability of Language Model Performance
  7. 2026What do Language Models Learn and When? The Implicit Curriculum Hypothesis
  8. 2025Can an Easy-to-Hard Curriculum Make Reasoning Emerge in Small Language Models? Evidence from a Four-Stage Curriculum on GPT-2