mlsanityData to 19 Oct 2026

Data mixtures and data selection

98 papers1 this weekHigh confidence in this grouping: the papers form one clear topicHigh confidence in this grouping: the papers form one clear topic · About

Keywords: selection, training, mixture, pretraining, performance

Key facts. Data mixtures and data selection: 98 papers on arXiv since 2021 (data through 19 Oct 2026). 43 of its 98 mapped papers (44%) cite “DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining” (2023, arXiv 2305.10429). 4 new papers 1 Oct–19 Oct.

What’s new

1 Oct – 19 Oct 2026

  1. This week · 1
  2. ICLHerd: Herding-Based Data Selection for LLMs with In-Context Learning as a Proxy
  3. Earlier · 3
  4. Measuring Data Diversity with Embeddings: A Taxonomy and Benchmark for Diversity Measures also fits: Pre-training corpora and quality filtering
  5. ZOIF: Zeroth-Order Influence Function Estimation for LLM Data Selection Using Only Forward Passes
  6. Host-Aware Web Crawling for LLM Pretraining Data also fits: Pre-training corpora and quality filtering

Foundations

Most cited by papers here

  1. DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining2023cited by 43 here
  2. Data Selection for Language Models via Importance Resampling2023cited by 37 here
  3. RegMix: Data Mixture as Regression for Language Model Pre-training2024cited by 28 here
  4. Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance2024cited by 23 here
  5. SemDeDup: Data-efficient learning at web-scale through semantic deduplication2023cited by 22 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 42 here
  2. 2021Deduplicating Training Data Makes Language Models Bettercited by 21 here
  3. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 25 here
  4. 2022Training Compute-Optimal Large Language Modelscited by 25 here
  5. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 16 here
  6. 2023Data Selection for Language Models via Importance Resamplingcited by 37 here
  7. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 43 here
  8. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 16 here
  9. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 18 here
  10. 2024A Survey on Data Selection for Language Modelscited by 11 here
  11. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 14 here
  12. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 28 here
  13. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 9 here
11 more it builds on, not in this chain
  1. 2020Scaling Laws for Neural Language Modelscited by 25 here
  2. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 11 here
  3. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 22 here
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 14 here
  5. 2023Scaling Data-Constrained Language Modelscited by 17 here
  6. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 16 here
  7. 2023Efficient Online Data Mixing For Language Model Pre-Trainingcited by 18 here
  8. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 23 here
  9. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 23 here
  10. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 24 here
  11. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 18 here

The 24 most cited of 105.

Browse

30 of 98, ranked 6–35

  1. 6Deduplicating Training Data Makes Language Models Better202121
  2. 7QuRating: Selecting High-Quality Data for Training Language Models202418
  3. 8Efficient Online Data Mixing For Language Model Pre-Training202318
  4. 9D4: Improving LLM Pretraining via Document De-Duplication and Diversification202316
  5. 10MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models202414
  6. 11A Survey on Data Selection for Language Models202411
  7. 12Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnostic20247
  8. 13QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM Pretraining20256
  9. 14BiMix: A Bivariate Data Mixing Law for Language Model Pretraining20246
  10. 15AutoScale: Scale-Aware Data Mixing for Pre-Training LLMs20246
All 30
  1. 16SlimPajama-DC: Understanding Data Combinations for LLM Training20236
  2. 17Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training20255
  3. 18Scaling Laws for Optimal Data Mixtures20255
  4. 19Self-Influence Guided Data Reweighting for Language Model Pre-training20235
  5. 20Harnessing Diversity for Important Data Selection in Pretraining Large Language Models20244
  6. 21Data, Data Everywhere: A Guide for Pretraining Dataset Construction20243
  7. 22Instruction Mining: Instruction Data Selection for Tuning Large Language Models20233
  8. 23Group-Level Data Selection for Efficient Pretraining20252
  9. 24BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretraining20252
  10. 25CoLoR-Filter: Conditional Loss Reduction Filtering for Targeted Language Model Pre-training20242
  11. 26CMR Scaling Law: Predicting Critical Mixture Ratios for Continual Pre-training of Language Models20242
  12. 27Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scale20242
  13. 28Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretraining20242
  14. 29OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration20261
  15. 30Data Mixing for Large Language Models Pretraining: A Survey and Outlook20261
  16. 31InfoLaw: Information Scaling Laws for Large Language Models with Quality-Weighted Mixture Data and Repetition20261
  17. 32Scaling Laws for Mixture Pretraining Under Data Constraints20261
  18. 33FOLD: Fuzzy Online Deduplication for Very Large Evolving Datasets via Approximate Nearest Neighbor Search20261
  19. 34Optimizing Pretraining Data Mixtures with LLM-Estimated Utility20251
  20. 35Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining20251