# Pre-training objectives and curricula

Area: [LLM pre-training data and scaling laws](https://mlsanity.com/pretrain/index.md). HTML: https://mlsanity.com/pretrain/pre-training-objectives-and-curricula/

## Key facts

- Pre-training objectives and curricula: 61 papers on arXiv since 2020 (data through 19 Oct 2026).
- 5 of its 61 mapped papers (8%) cite “Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning” (2025, arXiv 2506.11300).
- 3 of its 61 mapped papers (5%) cite “UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training” (2020, arXiv 2002.12804).
- 3 new papers 1 Oct–19 Oct.

## Most cited here

- [Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning](https://mlsanity.com/paper/2506.11300/) (2025), cited by 5 papers here
- [UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training](https://mlsanity.com/paper/2002.12804/) (2020), cited by 3 papers here
- [Knowledge-Aware Language Model Pretraining](https://mlsanity.com/paper/2007.00655/) (2020), cited by 3 papers here
- [COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining](https://mlsanity.com/paper/2102.08473/) (2021), cited by 3 papers here
- [Curriculum learning for language modeling](https://mlsanity.com/paper/2108.02170/) (2021), cited by 3 papers here
- [Dict-BERT: Enhancing Language Model Pre-training with Dictionary](https://mlsanity.com/paper/2110.06490/) (2021), cited by 2 papers here
- [What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?](https://mlsanity.com/paper/2204.05832/) (2022), cited by 2 papers here
- [InforMask: Unsupervised Informative Masking for Language Model Pretraining](https://mlsanity.com/paper/2210.11771/) (2022), cited by 2 papers here
- [Irreducible Curriculum for Language Model Pretraining](https://mlsanity.com/paper/2310.15389/) (2023), cited by 2 papers here
- [Metadata Conditioning Accelerates Language Model Pre-training](https://mlsanity.com/paper/2501.01956/) (2025), cited by 2 papers here

## New (1 Oct–19 Oct)

- [Learning to Read Out: Unembedding Dynamics in Language Model Pretraining](https://mlsanity.com/paper/2610.21025/) (2026-10-14)
- [Language Exposure Matters: Distribution and Order in Multilingual BabyLM Pre-training](https://mlsanity.com/paper/2610.20102/) (2026-10-14)
- [Latent Taxonomies: How Property Induction Develops in Language Model Pre-training](https://mlsanity.com/paper/2610.15836/) (2026-10-11)