mlsanityData to 14 Oct 2026

Parameterized Synthetic Text Generation with SimpleStories

arXiv 2504.09184 ↗First posted 12 Apr 2025cs.CL cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

We present SimpleStories, a large synthetic story dataset in simple language, consisting of 2 million samples each in English and Japanese. Through parameterizing prompts at multiple levels of abstraction, we achieve control over story characteristics at scale, inducing syntactic and semantic diversity. Ablations on a newly trained model suite show improved sample efficiency and model interpretability compared to the TinyStories dataset. We open-source all constituent parts of model creation, hoping to enable novel ways to study the end-to-end training process. As a byproduct, we move the frontier regarding the fewest-parameter language model that outputs grammatical natural language.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 17 of its 27 references that are in the map. First posted on arXiv 12 Apr 2025.

Builds on

17 of 27 references.

References · 17

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  6. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  7. 2018Sanity Checks for Saliency Mapscited by 1,156
  8. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  9. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  10. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  11. 20242 OLMo 2 Furiouscited by 340
  12. 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 88
5 more
  1. 2024Detection and Measurement of Syntactic Templates in Generated Textcited by 44
  2. 2024Reconciling Kaplan and Chinchilla Scaling Lawscited by 39
  3. 2024Bilinear MLPs enable weight-based mechanistic interpretabilitycited by 28
  4. 2020fugashi, a Tool for Tokenizing Japanese in Pythoncited by 17
  5. 2024Super Tiny Language Modelscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026Alignment Whack-a-Mole : Finetuning Activates Verbatim Recall of Copyrighted Books in Large Language Modelscited by 6
  2. 2026Annotations Mitigate Post-Training Mode Collapsecited by 5
  3. 2026Stories in Space: In-Context Learning Trajectories in Conceptual Belief Spacecited by 3
  4. 2026Modular Pretraining Enables Access Controlcited by 3
  5. 2026Sparse Crosscoders for diffing MoEs and Dense modelscited by 1
  6. 2026Spectral Reach: Understanding Neural Scaling as Progress into the Spectral Tailcited by 1
  7. 2026Cross-Lingual Stability of LLM Judges Under Controlled Generation: Evidence from Finno-Ugric Languagescited by 0
  8. 2026Towards Faster Language Model Inference Using Mixture-of-Experts Flow Matchingcited by 0
  9. 2026Sessa: Selective State Space Attentioncited by 0
  10. 2026Towards Human-Level Book-Writing Capabilitycited by 0