mlsanityData to 19 Oct 2026

LLM pre-training data and scaling laws

8 directions · 597 papers · 1 this week

Busiest this week

All directions

Largest first · week from Mon 19 Oct

  1. Optimizers and stability in LLM pre-trainingtraining, large, pre-training, languageHigh confidence in this grouping: the papers form one clear topic · About1130High confidence in this grouping: the papers form one clear topic
  2. Data mixtures and data selectionselection, training, mixture, pretrainingHigh confidence in this grouping: the papers form one clear topic · About981High confidence in this grouping: the papers form one clear topic
  3. Pre-training corpora and quality filteringlanguage, llms, languages, multilingualHigh confidence in this grouping: the papers form one clear topic · About960High confidence in this grouping: the papers form one clear topic
  4. Scaling laws and compute-optimal trainingscaling, laws, scaling laws, lawHigh confidence in this grouping: the papers form one clear topic · About870High confidence in this grouping: the papers form one clear topic
  5. Continual pre-training and training dynamicscontinual, pre-training, language, knowledgeHigh confidence in this grouping: the papers form one clear topic · About720High confidence in this grouping: the papers form one clear topic
  6. Pre-training objectives and curriculalanguage, pretraining, pre-training, knowledgeHigh confidence in this grouping: the papers form one clear topic · About610High confidence in this grouping: the papers form one clear topic
  7. Tokenization and byte-level modelstokenization, subword, language, tokenizerHigh confidence in this grouping: the papers form one clear topic · About510High confidence in this grouping: the papers form one clear topic
  8. Multi-token predictionprediction, multi-token, multi-token prediction, mtpHigh confidence in this grouping: the papers form one clear topic · About190High confidence in this grouping: the papers form one clear topic

Weekly activity

Papers per week, 6 Jul – 19 Oct (last week partial)

  1. DirectionConfidencepapers16 wks
  2. Optimizers and stability in LLM pre-trainingHigh confidence in this grouping: the papers form one clear topic · AboutHigh confidence in this grouping: the papers form one clear topicPer week: 6 Jul: 1; 13 Jul: 3; 20 Jul: 1; 27 Jul: 0; 3 Aug: 0; 10 Aug: 1; 17 Aug: 0; 24 Aug: 2; 31 Aug: 1; 7 Sep: 0; 14 Sep: 3; 21 Sep: 2; 28 Sep: 8; 5 Oct: 1; 12 Oct: 2; 19 Oct: 011325
  3. Data mixtures and data selectionHigh confidence in this grouping: the papers form one clear topic · AboutHigh confidence in this grouping: the papers form one clear topicPer week: 6 Jul: 0; 13 Jul: 0; 20 Jul: 0; 27 Jul: 1; 3 Aug: 1; 10 Aug: 2; 17 Aug: 0; 24 Aug: 1; 31 Aug: 0; 7 Sep: 0; 14 Sep: 0; 21 Sep: 1; 28 Sep: 1; 5 Oct: 2; 12 Oct: 1; 19 Oct: 19811
  4. Scaling laws and compute-optimal trainingHigh confidence in this grouping: the papers form one clear topic · AboutHigh confidence in this grouping: the papers form one clear topicPer week: 6 Jul: 1; 13 Jul: 0; 20 Jul: 0; 27 Jul: 1; 3 Aug: 1; 10 Aug: 0; 17 Aug: 1; 24 Aug: 0; 31 Aug: 0; 7 Sep: 0; 14 Sep: 0; 21 Sep: 1; 28 Sep: 2; 5 Oct: 2; 12 Oct: 0; 19 Oct: 0879
  5. Tokenization and byte-level modelsHigh confidence in this grouping: the papers form one clear topic · AboutHigh confidence in this grouping: the papers form one clear topicPer week: 6 Jul: 1; 13 Jul: 0; 20 Jul: 0; 27 Jul: 1; 3 Aug: 2; 10 Aug: 1; 17 Aug: 0; 24 Aug: 0; 31 Aug: 1; 7 Sep: 0; 14 Sep: 2; 21 Sep: 0; 28 Sep: 2; 5 Oct: 1; 12 Oct: 1; 19 Oct: 05112
  6. Continual pre-training and training dynamicsHigh confidence in this grouping: the papers form one clear topic · AboutHigh confidence in this grouping: the papers form one clear topicPer week: 6 Jul: 0; 13 Jul: 0; 20 Jul: 0; 27 Jul: 0; 3 Aug: 0; 10 Aug: 0; 17 Aug: 1; 24 Aug: 0; 31 Aug: 1; 7 Sep: 0; 14 Sep: 1; 21 Sep: 1; 28 Sep: 3; 5 Oct: 0; 12 Oct: 0; 19 Oct: 0727
  7. Pre-training objectives and curriculaHigh confidence in this grouping: the papers form one clear topic · AboutHigh confidence in this grouping: the papers form one clear topicPer week: 6 Jul: 0; 13 Jul: 0; 20 Jul: 0; 27 Jul: 0; 3 Aug: 0; 10 Aug: 1; 17 Aug: 1; 24 Aug: 1; 31 Aug: 0; 7 Sep: 1; 14 Sep: 0; 21 Sep: 0; 28 Sep: 0; 5 Oct: 1; 12 Oct: 2; 19 Oct: 0617
  8. Pre-training corpora and quality filteringHigh confidence in this grouping: the papers form one clear topic · AboutHigh confidence in this grouping: the papers form one clear topicPer week: 6 Jul: 0; 13 Jul: 1; 20 Jul: 0; 27 Jul: 0; 3 Aug: 0; 10 Aug: 0; 17 Aug: 0; 24 Aug: 0; 31 Aug: 0; 7 Sep: 0; 14 Sep: 0; 21 Sep: 0; 28 Sep: 1; 5 Oct: 1; 12 Oct: 0; 19 Oct: 0963
  9. Multi-token predictionHigh confidence in this grouping: the papers form one clear topic · AboutHigh confidence in this grouping: the papers form one clear topicPer week: 6 Jul: 0; 13 Jul: 0; 20 Jul: 0; 27 Jul: 1; 3 Aug: 1; 10 Aug: 0; 17 Aug: 0; 24 Aug: 0; 31 Aug: 0; 7 Sep: 0; 14 Sep: 0; 21 Sep: 0; 28 Sep: 0; 5 Oct: 1; 12 Oct: 1; 19 Oct: 0194