mlsanityData to 7 Oct 2026

QuRating: Selecting High-Quality Data for Training Language Models

arXiv 2402.09739 ↗First posted 15 Feb 2024cs.CL cs.LGcited by 153

Not on the map. Why?

Abstract (arXiv)

Selecting high-quality pre-training data is important for creating capable language models, but existing methods rely on simple heuristics. We introduce QuRating, a method for selecting pre-training data that can capture human intuitions about data quality. In this paper, we investigate four qualities - writing style, required expertise, facts & trivia, and educational value - and find that LLMs are able to discern these qualities, especially when making pairwise judgments of texts. We train a QuRater model to learn scalar ratings from pairwise judgments, and use it to annotate a 260B training corpus with quality ratings for each of the four criteria. In our experiments, we select 30B tokens according to the different quality ratings and train 1.3B-parameter language models on the selected data. We find that it is important to balance quality and diversity. When we sample using quality ratings as logits over documents, our models obtain lower perplexity and stronger in-context learning performance than baselines. Our best model is based on educational value and performs similarly to a model trained with uniform sampling for 50% more steps. Beyond data selection, we use the quality ratings to construct a training curriculum which improves performance without changing the training dataset. We extensively analyze the quality ratings and discuss their characteristics, biases, and wider implications.

Key facts. Cited by 153 papers in the map (data through 7 Oct 2026). Builds on 41 of its 92 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

41 of 92 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,927
29 more
  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  3. 2020GLU Variants Improve Transformercited by 1,724
  4. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  5. 2023PaLM 2 Technical Reportcited by 1,154
  6. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  7. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  8. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  9. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  10. 2023Textbooks Are All You Needcited by 561
  11. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  12. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  13. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  14. 2023Scaling Data-Constrained Language Modelscited by 391
  15. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  16. 2016FastText.zip: Compressing text classification modelscited by 331
  17. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  18. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
  19. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  20. 2023Pretraining Language Models with Human Preferencescited by 245
  21. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 146
  22. 2019Assessing Social and Intersectional Biases in Contextualized Word Representationscited by 134
  23. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 118
  24. 2023A Theory for Emergence of Complex Skills in Language Modelscited by 111
  25. 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 80
  26. 2023Skill-Mix: a Flexible and Expandable Family of Evaluations for AI modelscited by 64
  27. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 59
  28. 2024AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filterscited by 21
  29. 2023When Less is More: Investigating Data Pruning for Pretraining LLMs at Scalecited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 26; 2025: 82; 2026: 45.

153 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Models for Code Generationcited by 646
  2. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  3. 2024An Introduction to Vision-Language Modelingcited by 175
  4. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  5. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  6. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 96
  7. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 73
  8. 2025Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-trainingcited by 62
  9. 2024Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating Worldcited by 49
  10. 2025Scaling Laws for Optimal Data Mixturescited by 49
  11. 2024Entropy Law: The Story Behind Data Compression and LLM Performancecited by 42
  12. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
38 more
  1. 2024Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scalecited by 37
  2. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examplescited by 37
  3. 2025Ultra-FineWeb: Efficient Data Filtering and Verification for High-Quality LLM Training Datacited by 35
  4. 2024Harnessing Diversity for Important Data Selection in Pretraining Large Language Modelscited by 34
  5. 2025Large-Scale Data Selection for Instruction Tuningcited by 34
  6. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  7. 2025Enhancing Multilingual LLM Pretraining with Model-Based Data Selectioncited by 33
  8. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
  9. 2025Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Modelscited by 31
  10. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  11. 2025Shifting AI Efficiency From Model-Centric to Data-Centric Compressioncited by 29
  12. 2024TAGCOS: Task-agnostic Gradient Clustered Coreset Selection for Instruction Tuning Datacited by 28
  13. 2024ProSec: Fortifying Code LLMs with Proactive Security Alignmentcited by 24
  14. 2025DataFlow: An LLM-Driven Framework for Unified Data Preparation and Workflow Automation in the Era of Data-Centric AIcited by 22
  15. 2024Weak-to-Strong Generalization Through the Data-Centric Lenscited by 21
  16. 2025Metadata Conditioning Accelerates Language Model Pre-trainingcited by 20
  17. 2025Predictive Data Selection: The Data That Predicts Is the Data That Teachescited by 20
  18. 2025Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Modelscited by 19
  19. 2025Language Models Improve When Pretraining Data Matches Target Taskscited by 18
  20. 2025A Survey on LLM Mid-Trainingcited by 18
  21. 2026OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iterationcited by 17
  22. 2025Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretrainingcited by 16
  23. 2025QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM Pretrainingcited by 16
  24. 2024ScalingFilter: Assessing Data Quality through Inverse Utilization of Scaling Lawscited by 15
  25. 2025RefineX: Learning to Refine Pre-training Data at Scale from Expert-Guided Programscited by 15
  26. 2025Token Weighting for Long-Range Language Modelingcited by 14
  27. 2025Utility-Diversity Aware Online Batch Selection for LLM Supervised Fine-tuningcited by 14
  28. 2024Scalable Fine-tuning from Multiple Data Sources: A First-Order Approximation Approachcited by 13
  29. 2025Diversity as a Reward: Fine-Tuning LLMs on a Mixture of Domain-Undetermined Datacited by 13
  30. 2025Group-Level Data Selection for Efficient Pretrainingcited by 12
  31. 2025SynthRL: Scaling Visual Reasoning with Verifiable Data Synthesiscited by 12
  32. 2025How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretrainingcited by 12
  33. 2024FANNO: Augmenting High-Quality Instruction Data with Open-Sourced LLMs Onlycited by 11
  34. 2025Dataset Protection via Watermarked Canaries in Retrieval-Augmented LLMscited by 11
  35. 2024Efficient Pretraining Data Selection for Language Models via Multi-Actor Collaborationcited by 10
  36. 2024Compute-Constrained Data Selectioncited by 10
  37. 2025Quality-Driven Curation of Remote Sensing Vision-Language Data via Learned Scoring Modelscited by 10
  38. 2025SampleMix: A Sample-wise Pre-training Data Mixing Strategey by Coordinating Data Quality and Diversitycited by 10

50 most cited of 153 All 153 (JSON)