mlsanityData to 13 Oct 2026

LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure

arXiv 2608.13545 ↗First posted 13 Aug 2026cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Modern language models are trained on heterogeneous web-scale text corpora. Consequently, studying knowledge and skill acquisition is difficult, as prior exposure to related content is hard to characterize. To address this challenge, we introduce LittleCurriculum, a curated 88B-token pretraining corpus tailored to U.S. elementary school material, explicitly excluding concepts, facts, and vocabulary taught above Grade 5. Training a 5B-parameter LLM from scratch on LittleCurriculum yields LittleLearner, a model with sufficient language competence for open-ended evaluation, yet with clear knowledge and capability boundaries mapped to interpretable curriculum guidelines. We release LittleCurriculum and LittleLearner as a developmentally restricted sandbox to study how models acquire, represent, and use data under a well-defined training scope. We illustrate the sandbox's utility in a first suite of experiments on injecting new knowledge through post-training and in-context learning. These methods let LittleLearner better utilize existing knowledge, but do not raise out-of-scope capabilities. Our findings underscore the value of this controlled environment for future investigations.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 41 of its 63 references that are in the map. First posted on arXiv 13 Aug 2026.

Builds on

41 of 63 references.

References · 41

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,916
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,091
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,804
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,968
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,851
  6. 2022Language Models (Mostly) Know What They Knowcited by 1,823
  7. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,799
  8. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,424
  9. 2023LIMA: Less Is More for Alignmentcited by 1,091
  10. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 964
  11. 2016Bag of Tricks for Efficient Text Classificationcited by 872
  12. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 814
29 more
  1. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 544
  2. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 518
  3. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 507
  4. 2025Humanity's Last Examcited by 478
  5. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 404
  6. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 244
  7. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 194
  8. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 151
  9. 2023Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentationcited by 146
  10. 2022Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)cited by 137
  11. 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 137
  12. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretrainingcited by 125
  13. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 120
  14. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scalingcited by 113
  15. 2024LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancementcited by 69
  16. 2025On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Modelscited by 67
  17. 2023Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpuscited by 62
  18. 2021Word Acquisition in Neural Language Modelscited by 41
  19. 2024Procedural Knowledge in Pretraining Drives Reasoning in Large Language Modelscited by 41
  20. 2024Benchmarking Knowledge Boundary for Large Language Models: A Different Perspective on Model Evaluationcited by 38
  21. 2024Knowledge Boundary of Large Language Models: A Surveycited by 38
  22. 2024Training on the Test Task Confounds Evaluation and Emergencecited by 34
  23. 2024In Search of Forgotten Domain Generalizationcited by 21
  24. 2025LLMs on the Line: Data Determines Loss-to-Loss Scaling Lawscited by 16
  25. 2025MATH-Beyond: A Benchmark for RL to Expand Beyond the Base Modelcited by 10
  26. 2024From Next-Token to Mathematics: The Learning Dynamics of Mathematical Reasoning in Language Modelscited by 6
  27. 2026Operationalising the Superficial Alignment Hypothesis via Task Complexitycited by 4
  28. 2026Beyond Alignment: Expanding Reasoning Capacity via Manifold-Reshaping Policy Optimizationcited by 3
  29. 2026Knowing Isn't Understanding: Re-grounding Generative Proactivity with Epistemic and Behavioral Insightcited by 2

Built on by

Not cited yet.