mlsanityData to 19 Oct 2026

Pre-training objectives and curricula

61 papers0 this weekHigh confidence in this grouping: the papers form one clear topicHigh confidence in this grouping: the papers form one clear topic · About

Keywords: language, pretraining, pre-training, knowledge, masking

Key facts. Pre-training objectives and curricula: 61 papers on arXiv since 2020 (data through 19 Oct 2026). 5 of its 61 mapped papers (8%) cite “Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning” (2025, arXiv 2506.11300). 3 new papers 1 Oct–19 Oct.

What’s new

1 Oct – 19 Oct 2026

3 new since 1 Oct

  1. Learning to Read Out: Unembedding Dynamics in Language Model Pretraining also fits: Continual pre-training and training dynamics
  2. Language Exposure Matters: Distribution and Order in Multilingual BabyLM Pre-training also fits: Continual pre-training and training dynamics
  3. Latent Taxonomies: How Property Induction Develops in Language Model Pre-training also fits: Continual pre-training and training dynamics

Foundations

Most cited by papers here

  1. Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning2025cited by 5 here
  2. UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training2020cited by 3 here
  3. Knowledge-Aware Language Model Pretraining2020cited by 3 here
  4. COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining2021cited by 3 here
  5. Curriculum learning for language modeling2021cited by 3 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 7 here
  2. 2021Deduplicating Training Data Makes Language Models Bettercited by 3 here
  3. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 6 here
  4. 2022Training Compute-Optimal Large Language Modelscited by 6 here
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 4 here
  6. 2023Scaling Data-Constrained Language Modelscited by 3 here
  7. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 3 here
  8. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 5 here
  9. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 3 here
15 more it builds on, not in this chain
  1. 2020Scaling Laws for Neural Language Modelscited by 15 here
  2. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 3 here
  3. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 7 here
  4. 2020Knowledge-Aware Language Model Pretrainingcited by 3 here
  5. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 3 here
  6. 2021Curriculum learning for language modelingcited by 3 here
  7. 2021The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Modelscited by 4 here
  8. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 3 here
  9. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 6 here
  10. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 3 here
  11. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 3 here
  12. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 6 here
  13. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 7 here
  14. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 4 here
  15. 2025Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learningcited by 5 here

The 24 most cited of 75.

Browse

30 of 61, ranked 6–35

  1. 6Metadata Conditioning Accelerates Language Model Pre-training20252
  2. 7Irreducible Curriculum for Language Model Pretraining20232
  3. 8What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?20222
  4. 9InforMask: Unsupervised Informative Masking for Language Model Pretraining20222
  5. 10Dict-BERT: Enhancing Language Model Pre-training with Dictionary20212
  6. 11When Does Metadata Conditioning (NOT) Work for Language Model Pre-Training? A Study with Context-Free Grammars20251
  7. 12Rethinking the Role of Text Complexity in Language Model Pretraining20251
  8. 13Accelerating Large Language Model Pretraining via LFR Pedagogy: Learn, Focus, and Review20241
  9. 14UL2: Unifying Language Learning Paradigms20221
  10. 15On the Role of Bidirectionality in Language Model Pre-Training20221
All 30
  1. 16Learning Better Masking for Better Language Model Pre-training20221
  2. 17Effective Pre-Training Objectives for Transformer-based Autoencoders20221
  3. 18GLM: General Language Model Pretraining with Autoregressive Blank Infilling20211
  4. 19Curriculum Learning for LLM Pretraining: An Analysis of Learning Dynamics20260
  5. 20KoCo: Conditioning Language Model Pre-training on Knowledge Coordinates20260
  6. 21A Causal Language Modeling Detour Improves Encoder Continued Pretraining20260
  7. 22Pretraining Objective Matters in Extreme Low-Data FGVC: A Backbone-Controlled Study20260
  8. 23Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws20260
  9. 24Demystifying Training-Time Augmentation for Data-Constrained Language Model Pretraining20260
  10. 25LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure20260
  11. 26Understanding Curriculum Learning in Large Language Models via Cross-Difficulty Optimization Dynamics20260
  12. 27Unfolding Scientific Papers into Multi-Turn Generation Trajectories for Continued Pre-Training20260
  13. 28POSPAN: Position-Constrained Span Masking for Language Model Pre-training20260
  14. 29Scaling LLM Pre-training with Vocabulary Curriculum20250
  15. 30AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong Pretraining Data Selection20250
  16. 31Drop Dropout on Single-Epoch Language Model Pretraining20250
  17. 32Curriculum-Guided Layer Scaling for Language Model Pretraining20250
  18. 33Influence-driven Curriculum Learning for Pre-training on Limited Data20250
  19. 34Patent Language Model Pretraining with ModernBERT20250
  20. 35Beyond Repetition: Text Simplification and Curriculum Learning for Data-Constrained Pretraining20250