mlsanityData to 13 Oct 2026

DataRater: Meta-Learned Dataset Curation

arXiv 2505.17895 ↗First posted 23 May 2025stat.ML cs.AI cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

The quality of foundation models depends heavily on their training data. Consequently, great efforts have been put into dataset curation. Yet most approaches rely on manual tuning of coarse-grained mixtures of large buckets of data, or filtering by hand-crafted heuristics. An approach that is ultimately more scalable (let alone more satisfying) is to learn which data is actually valuable for training. This type of meta-learning could allow more sophisticated, fine-grained, and effective curation. Our proposed DataRater is an instance of this idea. It estimates the value of training on any particular data point. This is done by meta-learning using `meta-gradients', with the objective of improving training efficiency on held out data. In extensive experiments across a range of model scales and datasets, we find that using our DataRater to filter data is highly effective, resulting in significantly improved compute efficiency.

Key facts. Cited by 13 papers in the map (data through 13 Oct 2026). Builds on 41 of its 65 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

41 of 65 references.

References · 41

  1. 2017Attention Is All You Needcited by 60,106
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,231
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,798
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,053
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,843
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,379
  9. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,805
  10. 2018DARTS: Differentiable Architecture Searchcited by 1,789
  11. 2017Understanding Black-box Predictions via Influence Functionscited by 1,729
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,201
29 more
  1. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,126
  2. 2023LIMA: Less Is More for Alignmentcited by 1,091
  3. 2016Bag of Tricks for Efficient Text Classificationcited by 872
  4. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 621
  5. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 562
  6. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  7. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 477
  8. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 391
  9. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 359
  10. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 303
  11. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 234
  12. 2016Hyperparameter optimization with approximate gradientcited by 232
  13. 2024A Survey on Data Selection for Language Modelscited by 232
  14. 2018Meta-Gradient Reinforcement Learningcited by 158
  15. 2022A framework for bilevel optimization that enables stochastic and global variance reduction algorithmscited by 99
  16. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 84
  17. 2024Data Shapley in One Training Runcited by 79
  18. 2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
  19. 2019Optimizing Data Usage via Differentiable Rewardscited by 48
  20. 2023Meta-Learning Online Adaptation of Language Modelscited by 43
  21. 2024Data Selection via Optimal Control for Language Modelscited by 38
  22. 2021Podracer architectures for scalable Reinforcement Learningcited by 37
  23. 2024ScaleBiO: Scalable Bilevel Optimization for LLM Data Reweightingcited by 35
  24. 2025Optimizing ML Training with Metagradient Descentcited by 23
  25. 2025Predictive Data Selection: The Data That Predicts Is the Data That Teachescited by 20
  26. 2025Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretrainingcited by 16
  27. 2021Robust Optimization for Multilingual Translation with Imbalanced Datacited by 15
  28. 2024Data, Data Everywhere: A Guide for Pretraining Dataset Constructioncited by 11
  29. 2025Scalable Meta-Learning via Mixed-Mode Differentiationcited by 4

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 5; 2026: 8.

13 papers.

Sort:

By area:

  1. 2025Language Self-Play For Data-Free Trainingcited by 42
  2. 2025Language Models Improve When Pretraining Data Matches Target Taskscited by 18
  3. 2026Towards Agentic Intelligence for Materials Sciencecited by 3
  4. 2026Synthetic Data for any Differentiable Targetcited by 2
  5. 2025Holdout-Loss-Based Data Selection for LLM Finetuning via In-Context Learningcited by 1
  6. 2025Alchemist: Unlocking Efficiency in Text-to-Image Model Training via Meta-Gradient Data Selectioncited by 1
  7. 2026Generalization Guarantees on Data-Driven Tuning of Gradient Descent with Langevin Updatescited by 1
  8. 2026NoiseRater: Meta-Learned Noise Valuation for Diffusion Model Trainingcited by 1
  9. 2025Uncovering Competency Gaps in Large Language Models and Their Benchmarkscited by 0
  10. 2026TBDFiltering: Sample-Efficient Tree-Based Data Filteringcited by 0
  11. 2026Tula: Optimizing Time, Cost, and Generalization in Distributed Large-Batch Trainingcited by 0
  12. 2026AMO: Operator-level Adaptive Muon Orthogonalizationcited by 0
1 more
  1. 2026Scalable, Transferable Meta-network for Data Selection Requires a Different Loss (and Why the Obvious Choice is Problematic)cited by 0