# LLM pre-training data and scaling laws

597 papers in 8 research directions. Data through 19 Oct 2026.
HTML: https://mlsanity.com/pretrain/

## Directions (largest first)

- [Optimizers and stability in LLM pre-training](https://mlsanity.com/pretrain/optimizers-and-stability-in-llm-pre-training/index.md): 113 papers, 0 new this week
- [Data mixtures and data selection](https://mlsanity.com/pretrain/data-mixtures-and-data-selection/index.md): 98 papers, 1 new this week
- [Pre-training corpora and quality filtering](https://mlsanity.com/pretrain/pre-training-corpora-and-quality-filtering/index.md): 96 papers, 0 new this week
- [Scaling laws and compute-optimal training](https://mlsanity.com/pretrain/scaling-laws-and-compute-optimal-training/index.md): 87 papers, 0 new this week
- [Continual pre-training and training dynamics](https://mlsanity.com/pretrain/continual-pre-training-and-training-dynamics/index.md): 72 papers, 0 new this week
- [Pre-training objectives and curricula](https://mlsanity.com/pretrain/pre-training-objectives-and-curricula/index.md): 61 papers, 0 new this week
- [Tokenization and byte-level models](https://mlsanity.com/pretrain/tokenization-and-byte-level-models/index.md): 51 papers, 0 new this week
- [Multi-token prediction](https://mlsanity.com/pretrain/multi-token-prediction/index.md): 19 papers, 0 new this week