# Data mixtures and data selection

Area: [LLM pre-training data and scaling laws](https://mlsanity.com/pretrain/index.md). HTML: https://mlsanity.com/pretrain/data-mixtures-and-data-selection/

## Key facts

- Data mixtures and data selection: 98 papers on arXiv since 2021 (data through 19 Oct 2026).
- 43 of its 98 mapped papers (44%) cite “DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining” (2023, arXiv 2305.10429).
- 37 of its 98 mapped papers (38%) cite “Data Selection for Language Models via Importance Resampling” (2023, arXiv 2302.03169).
- 4 new papers 1 Oct–19 Oct.

## Most cited here

- [DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining](https://mlsanity.com/paper/2305.10429/) (2023), cited by 43 papers here
- [Data Selection for Language Models via Importance Resampling](https://mlsanity.com/paper/2302.03169/) (2023), cited by 37 papers here
- [RegMix: Data Mixture as Regression for Language Model Pre-training](https://mlsanity.com/paper/2407.01492/) (2024), cited by 28 papers here
- [Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance](https://mlsanity.com/paper/2403.16952/) (2024), cited by 23 papers here
- [SemDeDup: Data-efficient learning at web-scale through semantic deduplication](https://mlsanity.com/paper/2303.09540/) (2023), cited by 22 papers here
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021), cited by 21 papers here
- [Efficient Online Data Mixing For Language Model Pre-Training](https://mlsanity.com/paper/2312.02406/) (2023), cited by 18 papers here
- [QuRating: Selecting High-Quality Data for Training Language Models](https://mlsanity.com/paper/2402.09739/) (2024), cited by 18 papers here
- [D4: Improving LLM Pretraining via Document De-Duplication and Diversification](https://mlsanity.com/paper/2308.12284/) (2023), cited by 16 papers here
- [MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models](https://mlsanity.com/paper/2406.06046/) (2024), cited by 14 papers here

## New (1 Oct–19 Oct)

- [ICLHerd: Herding-Based Data Selection for LLMs with In-Context Learning as a Proxy](https://mlsanity.com/paper/2610.27142/) (2026-10-19)
- [Measuring Data Diversity with Embeddings: A Taxonomy and Benchmark for Diversity Measures](https://mlsanity.com/paper/2610.24888/) (2026-10-17)
- [ZOIF: Zeroth-Order Influence Function Estimation for LLM Data Selection Using Only Forward Passes](https://mlsanity.com/paper/2610.15336/) (2026-10-11)
- [Host-Aware Web Crawling for LLM Pretraining Data](https://mlsanity.com/paper/2610.13321/) (2026-10-09)