mlsanityData to 14 Oct 2026

Data Selection via Optimal Control for Language Models

arXiv 2410.07064 ↗First posted 9 Oct 2024cs.CLcited by 38

Not on the map. Why?

Abstract (arXiv)

This work investigates the selection of high-quality pre-training data from massive corpora to enhance LMs' capabilities for downstream usage. We formulate data selection as a generalized Optimal Control problem, which can be solved theoretically by Pontryagin's Maximum Principle (PMP), yielding a set of necessary conditions that characterize the relationship between optimal data selection and LM training dynamics. Based on these theoretical results, we introduce PMP-based Data Selection (PDS), a framework that approximates optimal data selection by solving the PMP conditions. In our experiments, we adopt PDS to select data from CommmonCrawl and show that the PDS-selected corpus accelerates the learning of LMs and constantly boosts their performance on a wide range of downstream tasks across various model sizes. Moreover, the benefits of PDS extend to ~400B models trained on ~10T tokens, as evidenced by the extrapolation of the test loss curves according to the Scaling Laws. PDS also improves data utilization when the pre-training data is limited, by reducing the data demand by 1.8 times, which helps mitigate the quick exhaustion of available web-crawled corpora. Our code, model, and data can be found at https://github.com/microsoft/LMOps/tree/main/data_selection.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Builds on 66 of its 87 references that are in the map. First posted on arXiv 9 Oct 2024.

Builds on

66 of 87 references.

References · 66

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  12. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
38 more
  1. 2023Mistral 7Bcited by 3,164
  2. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  6. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  7. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  8. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  9. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  10. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  11. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  12. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  13. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  14. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  15. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  16. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
  17. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  18. 2023Scaling Data-Constrained Language Modelscited by 403
  19. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 399
  20. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  21. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  22. 2023Studying Large Language Model Generalization with Influence Functionscited by 326
  23. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  24. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  25. 2016Data Programming: Creating Large Training Sets, Quicklycited by 246
  26. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
  27. 2024A Survey on Data Selection for Language Modelscited by 236
  28. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  29. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 220
  30. 2019You Only Propagate Once: Accelerating Adversarial Training via Maximal Principlecited by 183
  31. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  32. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 150
  33. 2024Rho-1: Not All Tokens Are What You Needcited by 133
  34. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 121
  35. 2023MoDS: Model-oriented Data Selection for Instruction Tuningcited by 107
  36. 2024DsDm: Model-Aware Dataset Selection with Datamodelscited by 105
  37. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 100
  38. 2023DoGE: Domain Reweighting with Generalization Estimationcited by 90

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 23; 2026: 13.

38 papers.

Sort:

By area:

  1. 2024NVILA: Efficient Frontier Visual Language Modelscited by 209
  2. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Modelscited by 29
  3. 2025Language Models Improve When Pretraining Data Matches Target Taskscited by 19
  4. 2025QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM Pretrainingcited by 16
  5. 2025How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretrainingcited by 14
  6. 2025Group-Level Data Selection for Efficient Pretrainingcited by 13
  7. 2025DataRater: Meta-Learned Dataset Curationcited by 13
  8. 2025Data-Efficient RLVR via Off-Policy Influence Guidancecited by 12
  9. 2025Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Schedulercited by 12
  10. 2026Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuningcited by 10
  11. 2025IDEAL: Data Equilibrium Adaptation for Multi-Capability Language Model Alignmentcited by 8
  12. 2025Data Efficacy for Language Model Trainingcited by 8
26 more
  1. 2026Optimsyn: Influence-Guided Rubrics Optimization for Synthetic Data Generationcited by 6
  2. 2025Domain2Vec: Vectorizing Datasets to Find the Optimal Data Mixture without Trainingcited by 5
  3. 2025Shift is Good: Mismatched Data Mixing Improves Test Performancecited by 5
  4. 2026DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Modelscited by 5
  5. 2025MASS: Mathematical Data Selection via Skill Graphs for Pretraining Large Language Modelscited by 4
  6. 2025Nonparametric Data Attribution for Diffusion Modelscited by 4
  7. 2025Mining Useful General Data for Low-Resource Domain Adaptationcited by 4
  8. 2025TRIM: Token-wise Attention-Derived Saliency for Data-Efficient Instruction Tuningcited by 3
  9. 2026Towards Next-Generation LLM Training: From the Data-Centric Perspectivecited by 3
  10. 2026Demystifying Data Organization for Enhanced LLM Trainingcited by 3
  11. 2025LAMDAS: LLM as an Implicit Classifier for Domain-specific Data Selectioncited by 2
  12. 2025RL-Guided Data Selection for Language Model Finetuningcited by 2
  13. 2026Rethinking Data Mixing from the Perspective of Large Language Modelscited by 2
  14. 2025Unifying and Optimizing Data Values for Selection via Sequential Decision-Makingcited by 1
  15. 2025Measuring Fine-Grained Relatedness in Multitask Learning via Data Attributioncited by 1
  16. 2025Data Selection for LLM Alignment Using Fine-Grained Preferencescited by 1
  17. 2025Holdout-Loss-Based Data Selection for LLM Finetuning via In-Context Learningcited by 1
  18. 2025Alchemist: Unlocking Efficiency in Text-to-Image Model Training via Meta-Gradient Data Selectioncited by 1
  19. 2026Late-to-Early Training: LET LLMs Learn Earlier, So Faster and Bettercited by 1
  20. 2026What Really Improves Mathematical Reasoning: Structured Reasoning Signals Beyond Pure Codecited by 1
  21. 2026Capability Self-Assessment in Large Language Modelscited by 1
  22. 2026CircuitLens: Reasoning Circuits as Data Selection Signals for Reinforcement Learning with Verifiable Rewardscited by 1
  23. 2025Data-Efficient Training by Evolved Samplingcited by 0
  24. 2026AMO: Operator-level Adaptive Muon Orthogonalizationcited by 0
  25. 2026Epiplexity Guided Data Selection and Generation for Out-of-Distribution Generalizationcited by 0
  26. 2026Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Samplingcited by 0