mlsanityData to 7 Oct 2026

TinyStories: How Small Can Language Models Be and Still Speak Coherent English?

arXiv 2305.07759 ↗First posted 12 May 2023cs.CL cs.AI cs.LGcited by 425

Not on the map. Why?

Cited by 27 in Synthetic data and self-improvement for LLMs, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Language models (LMs) are powerful tools for natural language processing, but they often struggle to produce coherent and fluent text when they are small. Models with around 125M parameters such as GPT-Neo (small) or GPT-2 (small) can rarely generate coherent and consistent English text beyond a few words even after extensive training. This raises the question of whether the emergence of the ability to produce coherent English text only occurs at larger scales (with hundreds of millions of parameters or more) and complex architectures (with many layers of global attention). In this work, we introduce TinyStories, a synthetic dataset of short stories that only contain words that a typical 3 to 4-year-olds usually understand, generated by GPT-3.5 and GPT-4. We show that TinyStories can be used to train and evaluate LMs that are much smaller than the state-of-the-art models (below 10 million total parameters), or have much simpler architectures (with only one transformer block), yet still produce fluent and consistent stories with several paragraphs that are diverse and have almost perfect grammar, and demonstrate reasoning capabilities. We also introduce a new paradigm for the evaluation of language models: We suggest a framework which uses GPT-4 to grade the content generated by these models as if those were stories written by students and graded by a (human) teacher. This new paradigm overcomes the flaws of standard benchmarks which often requires the model's output to be very structures, and moreover provides a multidimensional score for the model, providing scores for different capabilities such as grammar, creativity and consistency. We hope that TinyStories can facilitate the development, analysis and research of LMs, especially for low-resource or specialized domains, and shed light on the emergence of language capabilities in LMs.

Key facts. Cited by 425 papers in the map (data through 7 Oct 2026). Builds on 24 of its 36 references that are in the map. First posted on arXiv 12 May 2023.

Builds on

24 of 36 references.

References · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  11. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  12. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
12 more
  1. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  2. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  3. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  4. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  5. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  6. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  7. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
  8. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  9. 2015Visualizing and Understanding Neural Models in NLPcited by 199
  10. 2022Vision Transformers provably learn spatial structurecited by 92
  11. 2023How Do Transformers Learn Topic Structure: Towards a Mechanistic Understandingcited by 85
  12. 2023What Matters In The Structured Pruning of Generative Language Models?cited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 41; 2024: 129; 2025: 125; 2026: 130.

425 papers.

Sort:

By area:

  1. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  2. 2023Textbooks Are All You Needcited by 561
  3. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  4. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  5. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
  6. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 264
  7. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
  8. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  9. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 156
  10. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 153
  11. 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Datasetcited by 143
  12. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Datacited by 143
38 more
  1. 2023Taken out of context: On measuring situational awareness in LLMscited by 131
  2. 2024InFoBench: Evaluating Instruction Following Ability in Large Language Modelscited by 128
  3. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agentscited by 123
  4. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  5. 2024Small Language Models: Survey, Measurements, and Insightscited by 109
  6. 2024LLM Dataset Inference: Did you train on my dataset?cited by 107
  7. 2024Probabilistic Inference in Language Models via Twisted Sequential Monte Carlocited by 85
  8. 2023Exposing Attention Glitches with Flip-Flop Language Modelingcited by 74
  9. 2024What Evidence Do Language Models Find Convincing?cited by 74
  10. 2025Self-Adapting Language Modelscited by 71
  11. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 69
  12. 2023TinyGSM: achieving >80% on GSM8k with small language modelscited by 68
  13. 2023Learning to Model the World with Languagecited by 66
  14. 2023Auto-Regressive Next-Token Predictors are Universal Learnerscited by 65
  15. 2024Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNscited by 65
  16. 2024Identifying Functionally Important Features with End-to-End Sparse Dictionary Learningcited by 64
  17. 2024Scalable Optimization in the Modular Normcited by 64
  18. 2023Contrasting Linguistic Patterns in Human and LLM-Generated News Textcited by 62
  19. 2024Strong Model Collapsecited by 62
  20. 2024CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Modelcited by 60
  21. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Modelscited by 56
  22. 2024Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating Worldcited by 49
  23. 2024Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 48
  24. 2025Diverse Preference Optimizationcited by 48
  25. 2025From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuitcited by 45
  26. 2024SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Modelscited by 43
  27. 2024Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verificationcited by 43
  28. 2025Reasoning to Learn from Latent Thoughtscited by 43
  29. 2023Efficient RLHF: Reducing the Memory Usage of PPOcited by 40
  30. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 40
  31. 2025Weight-sparse transformers have interpretable circuitscited by 40
  32. 2024Loss Landscape Degeneracy and Stagewise Development in Transformerscited by 39
  33. 2024Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficientcited by 39
  34. 2023Generating Faithful Synthetic Data with Large Language Models: A Case Study in Computational Social Sciencecited by 37
  35. 2024Linguistic Collapse: Neural Collapse in (Large) Language Modelscited by 36
  36. 2024AutoVCoder: A Systematic Framework for Automated Verilog Code Generation using LLMscited by 36
  37. 2024MELTing point: Mobile Evaluation of Language Transformerscited by 35
  38. 2024Anim-Director: A Large Multimodal Model Powered Agent for Controllable Animation Video Generationcited by 35

50 most cited of 425 All 425 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024From Babbling to Fluency: Evaluating the Evolution of Language Models in Terms of Human Language Acquisition
  2. 2026Little Brains, Big Feats: Exploring Compact Language Models
  3. 2023Baby's CoThought: Leveraging Large Language Models for Enhanced Reasoning in Compact Models
  4. 2026Towards the Holographic Characteristic of LLMs for Efficient Short-text Generation
  5. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
  6. 2026Break Step: Recursive Training Resonates with Replayed Sampling Noise
  7. 2025A Survey on Large Language Models with some Insights on their Capabilities and Limitations
  8. 2024Evaluating Language Models as Synthetic Data Generators