mlsanityData to 7 Oct 2026

Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models

arXiv 2405.20541 ↗First posted 30 May 2024cs.LG cs.CLcited by 83

Not on the map. Why?

Abstract (arXiv)

In this work, we investigate whether small language models can determine high-quality subsets of large-scale text datasets that improve the performance of larger language models. While existing work has shown that pruning based on the perplexity of a larger model can yield high-quality data, we investigate whether smaller models can be used for perplexity-based pruning and how pruning is affected by the domain composition of the data being pruned. We demonstrate that for multiple dataset compositions, perplexity-based pruning of pretraining data can significantly improve downstream task performance: pruning based on perplexities computed with a 125 million parameter model improves the average performance on downstream tasks of a 3 billion parameter model by up to 2.04 and achieves up to a 1.45× reduction in pretraining steps to reach commensurate baseline performance. Furthermore, we demonstrate that such perplexity-based data pruning also yields downstream performance gains in the over-trained and data-constrained regimes.

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Builds on 45 of its 60 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

45 of 60 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  11. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  12. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
33 more
  1. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  2. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  3. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  4. 2020Contrastive Learning with Hard Negative Samplescited by 619
  5. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  6. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  7. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  8. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 423
  9. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  10. 2020Hard Negative Mixing for Contrastive Learningcited by 399
  11. 2023Scaling Data-Constrained Language Modelscited by 391
  12. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  13. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 337
  14. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  15. 2021GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Trainingcited by 269
  16. 2023Data Filtering Networkscited by 247
  17. 2023Pretraining Language Models with Human Preferencescited by 245
  18. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
  19. 2018A Simple Method for Commonsense Reasoningcited by 223
  20. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  21. 2023Self-Alignment with Instruction Backtranslationcited by 149
  22. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  23. 2022DeepCore: A Comprehensive Library for Coreset Selection in Deep Learningcited by 146
  24. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 146
  25. 2024Language models scale reliably with over-training and on downstream taskscited by 105
  26. 2024Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnosticcited by 74
  27. 2024Smaller Language Models are capable of selecting Instruction-Tuning Training Data for Larger Language Modelscited by 32
  28. 2023Robust Data Pruning under Label Noise via Maximizing Re-labeling Accuracycited by 20
  29. 2023NLU on Data Diets: Dynamic Data Subset Selection for NLP Classification Taskscited by 19
  30. 2023LIMIT: Less Is More for Instruction Tuning Across Evaluation Paradigmscited by 14
  31. 2017Cynical Selection of Language Model Training Datacited by 13
  32. 2022Automatic Document Selection for Efficient Encoder Pretrainingcited by 11
  33. 2021An Empirical Exploration in Quality Filtering of Text Datacited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 45; 2026: 22.

83 papers.

Sort:

By area:

  1. 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Modelscited by 209
  2. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 73
  3. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Modelscited by 56
  4. 2025Stepwise Perplexity-Guided Refinement for Efficient Chain-of-Thought Reasoning in Large Language Modelscited by 51
  5. 2025The Best Instruction-Tuning Data are Those That Fitcited by 48
  6. 2025Diffusion Language Models are Super Data Learnerscited by 45
  7. 2024Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scalecited by 37
  8. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examplescited by 37
  9. 2024Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructionscited by 34
  10. 2025Large-Scale Data Selection for Instruction Tuningcited by 34
  11. 2025Enhancing Multilingual LLM Pretraining with Model-Based Data Selectioncited by 33
  12. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
38 more
  1. 2025SimPER: A Minimalist Approach to Preference Alignment without Hyperparameterscited by 25
  2. 2024How to Synthesize Text Data without Model Collapse?cited by 24
  3. 2025Improving Influence-based Instruction Tuning Data Selection for Balanced Learning of Diverse Capabilitiescited by 22
  4. 2024Weak-to-Strong Generalization Through the Data-Centric Lenscited by 21
  5. 2025Predictive Data Selection: The Data That Predicts Is the Data That Teachescited by 20
  6. 2025Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Modelscited by 19
  7. 2025Language Models Improve When Pretraining Data Matches Target Taskscited by 18
  8. 2026OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iterationcited by 17
  9. 2025Towards High Data Efficiency in Reinforcement Learning with Verifiable Rewardcited by 16
  10. 2024A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMscited by 15
  11. 2025Control the Temperature: Selective Sampling for Diverse and High-Quality LLM Outputscited by 15
  12. 2025Efficient Data Selection at Scale via Influence Distillationcited by 14
  13. 2025DataRater: Meta-Learned Dataset Curationcited by 13
  14. 2024Zyda-2: a 5 Trillion Token High-Quality Datasetcited by 12
  15. 2025Mitigating Forgetting in LLM Fine-Tuning via Low-Perplexity Token Learningcited by 12
  16. 2024Efficient Pretraining Data Selection for Language Models via Multi-Actor Collaborationcited by 10
  17. 2024Compute-Constrained Data Selectioncited by 10
  18. 2024Mastering Collaborative Multi-modal Data Selection: A Focus on Informativeness, Uniqueness, and Representativenesscited by 10
  19. 2025The Multilingual Divide and Its Impact on Global AI Safetycited by 10
  20. 2025Influence Functions for Efficient Data Selection in Reasoningcited by 10
  21. 2025HELIOS: Adaptive Model And Early-Exit Selection for Efficient LLM Inference Servingcited by 8
  22. 2025LearnAlign: Data Selection for LLM Reinforcement Learning with Improved Gradient Alignmentcited by 8
  23. 2025LLM Data Selection and Utilization via Dynamic Bi-level Optimizationcited by 7
  24. 2026GIST: Targeted Data Selection for Instruction Tuning via Coupled Optimization Geometrycited by 7
  25. 2024Language Model-Driven Data Pruning Enables Efficient Active Learningcited by 6
  26. 2024Not All LLM-Generated Data Are Equal: Rethinking Data Weighting in Text Classificationcited by 6
  27. 2025TiKMiX: Take Data Influence into Dynamic Mixture for Language Model Pre-trainingcited by 6
  28. 2025AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong Pretraining Data Selectioncited by 5
  29. 2025Domain2Vec: Vectorizing Datasets to Find the Optimal Data Mixture without Trainingcited by 5
  30. 2025A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthinesscited by 5
  31. 2025Improving Chemical Understanding of LLMs via SMILES Parsingcited by 4
  32. 2025A Survey of LLM × DATAcited by 4
  33. 2025Curriculum-Guided Layer Scaling for Language Model Pretrainingcited by 4
  34. 2025ADMIRE-BayesOpt: Accelerated Data MIxture RE-weighting for Language Models with Bayesian Optimizationcited by 4
  35. 2025Every Step Counts: Decoding Trajectories as Authorship Fingerprints of dLLMscited by 4
  36. 2026ReMiT: RL-Guided Mid-Training for Iterative LLM Evolutioncited by 4
  37. 2025MergeIT: From Selection to Merging for Efficient Instruction Tuningcited by 3
  38. 2025Prompts Generalize with Low Data: Non-vacuous Generalization Bounds for Optimizing Prompts with More Informative Priorscited by 3

50 most cited of 83 All 83 (JSON)