mlsanityData to 14 Oct 2026

Data Selection for Language Models via Importance Resampling

arXiv 2302.03169 ↗First posted 6 Feb 2023cs.CL cs.LGcited by 329

Not on the map. Why?

Cited by 20 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Selecting a suitable pretraining dataset is crucial for both general-domain (e.g., GPT-3) and domain-specific (e.g., Codex) language models (LMs). We formalize this problem as selecting a subset of a large raw unlabeled dataset to match a desired target distribution given unlabeled target samples. Due to the scale and dimensionality of the raw text data, existing methods use simple heuristics or require human experts to manually curate data. Instead, we extend the classic importance resampling approach used in low-dimensions for LM data selection. We propose Data Selection with Importance Resampling (DSIR), an efficient and scalable framework that estimates importance weights in a reduced feature space for tractability and selects data with importance resampling according to these weights. We instantiate the DSIR framework with hashed n-gram features for efficiency, enabling the selection of 100M documents from the full Pile dataset in 4.5 hours. To measure whether hashed n-gram features preserve the aspects of the data that are relevant to the target, we define KL reduction, a data metric that measures the proximity between the selected pretraining data and the target on some feature space. Across 8 data selection methods (including expert selection), KL reduction on hashed n-gram features highly correlates with average downstream accuracy (r=0.82). When selecting data for continued pretraining on a specific domain, DSIR performs comparably to expert curation across 8 target distributions. When pretraining general-domain models (target is Wikipedia and books), DSIR improves over random selection and heuristic filtering baselines by 2-2.5% on the GLUE benchmark. Code is available at https://github.com/p-lambda/dsir.

Key facts. Cited by 329 papers in the map (data through 14 Oct 2026). Builds on 59 of its 97 references that are in the map. First posted on arXiv 6 Feb 2023.

Builds on

59 of 97 references.

References · 59

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  11. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
38 more
  1. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  2. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  3. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  4. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  5. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  6. 2015Prioritized Experience Replaycited by 1,099
  7. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  8. 2016Bag of Tricks for Efficient Text Classificationcited by 876
  9. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  10. 2015Image-based Recommendations on Styles and Substitutescited by 706
  11. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  12. 2019Defending Against Neural Fake Newscited by 559
  13. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  14. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  15. 2019Quantifying the Carbon Emissions of Machine Learningcited by 521
  16. 2016Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filteringcited by 520
  17. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 377
  18. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 339
  19. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 328
  20. 2018Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Constructioncited by 322
  21. 2021GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Trainingcited by 274
  22. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
  23. 2020GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learningcited by 226
  24. 2019Stochastic Beams and Where to Find Them: The Gumbel-Top-k Trick for Sampling Sequences Without Replacementcited by 211
  25. 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 152
  26. 2020Understanding Self-Training for Gradual Domain Adaptationcited by 149
  27. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  28. 2020Cold-start Active Learning through Self-supervised Language Modelingcited by 128
  29. 2017PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstractscited by 124
  30. 2015Online Batch Selection for Faster Training of Neural Networkscited by 122
  31. 2020Telescoping Density-Ratio Estimationcited by 108
  32. 2021How to Train BERT with an Academic Budgetcited by 80
  33. 2021RETRIEVE: Coreset Selection for Efficient and Robust Semi-Supervised Learningcited by 74
  34. 2017Racial Disparity in Natural Language Processing: A Case Study of Social Media African-American Englishcited by 71
  35. 2019Reparameterizable Subset Sampling via Continuous Relaxationscited by 70
  36. 2022Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptationcited by 67
  37. 2017Learning to select data for transfer learning with Bayesian Optimizationcited by 64
  38. 2019Learning From Less Data: A Unified Data Subset Selection and Active Learning Framework for Computer Visioncited by 64

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 33; 2024: 104; 2025: 109; 2026: 83.

329 papers.

Sort:

By area:

  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  2. 2024A Survey on Large Language Models for Code Generationcited by 658
  3. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  4. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  5. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 399
  6. 2023SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generationcited by 386
  7. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  8. 2023Challenges and Applications of Large Language Modelscited by 254
  9. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
  10. 2023Efficient Large Language Models: A Surveycited by 186
  11. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 184
  12. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 182
38 more
  1. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 177
  2. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 157
  3. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 150
  4. 2024Generative Reward Modelscited by 148
  5. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 148
  6. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 145
  7. 2024Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Contentcited by 135
  8. 2024Rho-1: Not All Tokens Are What You Needcited by 133
  9. 2023CodeGen2: Lessons for Training LLMs on Programming and Natural Languagescited by 125
  10. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 121
  11. 2024DsDm: Model-Aware Dataset Selection with Datamodelscited by 105
  12. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learningcited by 104
  13. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 100
  14. 2024How to Train Data-Efficient LLMscited by 93
  15. 2023DoGE: Domain Reweighting with Generalization Estimationcited by 90
  16. 2023Anchor Points: Benchmarking Models with Much Fewer Examplescited by 86
  17. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 84
  18. 2023An Inverse Scaling Law for CLIP Trainingcited by 77
  19. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 76
  20. 2023The MiniPile Challenge for Data-Efficient Language Modelscited by 75
  21. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 75
  22. 2023SlimPajama-DC: Understanding Data Combinations for LLM Trainingcited by 72
  23. 2025Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-trainingcited by 66
  24. 2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
  25. 2023The Local Learning Coefficient: A Singularity-Aware Complexity Measurecited by 51
  26. 2025Infinity Instruct: Scaling Instruction Selection and Synthesis to Enhance Language Modelscited by 50
  27. 2024Improving Pretraining Data Using Perplexity Correlationscited by 46
  28. 2024Entropy Law: The Story Behind Data Compression and LLM Performancecited by 44
  29. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  30. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  31. 2024Loss Landscape Degeneracy and Stagewise Development in Transformerscited by 40
  32. 2024Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficientcited by 40
  33. 2025Tower+: Bridging Generality and Translation Specialization in Multilingual LLMscited by 40
  34. 2024Data Selection via Optimal Control for Language Modelscited by 38
  35. 2024ScaleBiO: Scalable Bilevel Optimization for LLM Data Reweightingcited by 37
  36. 2024Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scalecited by 37
  37. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 35
  38. 2025Enhancing Multilingual LLM Pretraining with Model-Based Data Selectioncited by 35

50 most cited of 329 All 329 (JSON)