mlsanityData to 7 Oct 2026

DsDm: Model-Aware Dataset Selection with Datamodels

arXiv 2401.12926 ↗First posted 23 Jan 2024cs.LG stat.MLcited by 103

Not on the map. Why?

Abstract (arXiv)

When selecting data for training large-scale models, standard practice is to filter for examples that match human notions of data quality. Such filtering yields qualitatively clean datapoints that intuitively should improve model behavior. However, in practice the opposite can often happen: we find that selecting according to similarity with "high quality" data sources may not increase (and can even hurt) performance compared to randomly selecting data. To develop better methods for selecting data, we start by framing dataset selection as an optimization problem that we can directly solve for: given target tasks, a learning algorithm, and candidate data, select the subset that maximizes model performance. This framework thus avoids handpicked notions of data quality, and instead models explicitly how the learning process uses train datapoints to predict on the target tasks. Our resulting method greatly improves language model (LM) performance on both pre-specified tasks and previously unseen tasks. Specifically, choosing target tasks representative of standard LM problems and evaluating on diverse held-out benchmarks, our selected datasets provide a 2x compute multiplier over baseline methods.

Key facts. Cited by 103 papers in the map (data through 7 Oct 2026). Builds on 49 of its 96 references that are in the map. First posted on arXiv 23 Jan 2024.

Builds on

49 of 96 references.

References · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  10. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
37 more
  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  3. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  4. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  5. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  7. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  8. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  9. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  10. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  11. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  12. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  13. 2018CoQA: A Conversational Question Answering Challengecited by 674
  14. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  15. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  16. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 337
  17. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  18. 2016NewsQA: A Machine Comprehension Datasetcited by 330
  19. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  20. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
  21. 2023TRAK: Attributing Model Behavior at Scalecited by 273
  22. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  23. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
  24. 2020GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learningcited by 224
  25. 2022Datamodels: Predicting Predictions from Training Datacited by 186
  26. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 120
  27. 2015The Goldilocks Principle: Reading Children's Books with Explicit Memory Representationscited by 108
  28. 2023Efficient Online Data Mixing For Language Model Pre-Trainingcited by 77
  29. 2021RETRIEVE: Coreset Selection for Efficient and Robust Semi-Supervised Learningcited by 72
  30. 2019Learning From Less Data: A Unified Data Subset Selection and Active Learning Framework for Computer Visioncited by 64
  31. 2016Coresets and Sketchescited by 63
  32. 2018A Swiss Army Infinitesimal Jackknifecited by 51
  33. 2019Optimizing Data Usage via Differentiable Rewardscited by 48
  34. 2023Repeated Random Sampling for Minimizing the Time-to-Accuracy of Learningcited by 26
  35. 2022Understanding Influence Functions and Datamodels via Harmonic Analysiscited by 23
  36. 2017Cynical Selection of Language Model Training Datacited by 13
  37. 2022Automatic Document Selection for Efficient Encoder Pretrainingcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 33; 2025: 44; 2026: 26.

103 papers.

Sort:

By area:

  1. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 172
  2. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  3. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 96
  4. 2024ContextCite: Attributing Model Generation to Contextcited by 93
  5. 2024What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functionscited by 88
  6. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 73
  7. 2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
  8. 2024Improving Pretraining Data Using Perplexity Correlationscited by 46
  9. 2024Adaptive Data Optimization: Dynamic Sample Selection with Scaling Lawscited by 46
  10. 2024Scalable Influence and Fact Tracing for Large Language Model Pretrainingcited by 40
  11. 2024Data Selection via Optimal Control for Language Modelscited by 38
  12. 2024Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scalecited by 37
38 more
  1. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examplescited by 37
  2. 2024Harnessing Diversity for Important Data Selection in Pretraining Large Language Modelscited by 34
  3. 2024Aioli: A Unified Optimization Framework for Language Model Data Mixingcited by 33
  4. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  5. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Modelscited by 27
  6. 2024Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedbackcited by 25
  7. 2025DataMIL: Selecting Data for Robot Imitation Learning with Datamodelscited by 25
  8. 2024Skywork-Math: Data Scaling Laws for Mathematical Reasoning in Large Language Models -- The Story Goes Oncited by 24
  9. 2024What Do Learning Dynamics Reveal About Generalization in LLM Reasoning?cited by 22
  10. 2025Improving Influence-based Instruction Tuning Data Selection for Balanced Learning of Diverse Capabilitiescited by 22
  11. 2024Stochastic Amortization: A Unified Approach to Accelerate Feature and Data Attributioncited by 21
  12. 2025CUPID: Curating Data your Robot Loves with Influence Functionscited by 21
  13. 2024textttdattri: A Library for Efficient Data Attributioncited by 18
  14. 2024Attribute-to-Delete: Machine Unlearning via Datamodel Matchingcited by 18
  15. 2024Loss-to-Loss Prediction: Scaling Laws for All Datasetscited by 15
  16. 2025Efficient Data Selection at Scale via Influence Distillationcited by 14
  17. 2025Diversity as a Reward: Fine-Tuning LLMs on a Mixture of Domain-Undetermined Datacited by 13
  18. 2025Angles Don't Lie: Unlocking Training-Efficient RL Through the Model's Own Signalscited by 13
  19. 2024Montessori-Instruct: Generate Influential Training Data Tailored for Student Learningcited by 12
  20. 2025Group-Level Data Selection for Efficient Pretrainingcited by 12
  21. 2025Data-Efficient RLVR via Off-Policy Influence Guidancecited by 12
  22. 2024Data, Data Everywhere: A Guide for Pretraining Dataset Constructioncited by 11
  23. 2024Efficient Pretraining Data Selection for Language Models via Multi-Actor Collaborationcited by 10
  24. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  25. 2026GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learningcited by 10
  26. 2024On Training Data Influence of GPT Modelscited by 9
  27. 2024TAROT: Targeted Data Selection via Optimal Transportcited by 9
  28. 2025The Singapore Consensus on Global AI Safety Research Prioritiescited by 9
  29. 2026Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuningcited by 9
  30. 2025IDEAL: Data Equilibrium Adaptation for Multi-Capability Language Model Alignmentcited by 8
  31. 2025Enhancing Training Data Attribution with Representational Optimizationcited by 8
  32. 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instancecited by 8
  33. 2024Influential Language Data Selection via Gradient Trajectory Pursuitcited by 7
  34. 2025Merge to Mix: Mixing Datasets via Model Mergingcited by 7
  35. 2025Fast Data Attribution for Text-to-Image Modelscited by 7
  36. 2024BaichuanSEED: Sharing the Potential of ExtensivE Data Collection and Deduplication by Introducing a Competitive Large Language Model Baselinecited by 6
  37. 2024Language Model-Driven Data Pruning Enables Efficient Active Learningcited by 6
  38. 2025Influence Guided Context Selection for Effective Retrieval-Augmented Generationcited by 6

50 most cited of 103 All 103 (JSON)