mlsanityData to 14 Oct 2026

Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling

arXiv 2410.03735 ↗First posted 30 Sep 2024cs.CL cs.LGcited by 18

Not on the map. Why?

Abstract (arXiv)

Specialist language models (LMs) focus on a specific task or domain on which they often outperform generalist LMs of the same size. However, the specialist data needed to pretrain these models is only available in limited amount for most tasks. In this work, we build specialist models from large generalist training sets instead. We propose a novel method, ClusteRed Importance SamPling (CRISP). CRISP clusters the generalist dataset and samples from these clusters based on their frequencies in the smaller specialist dataset. It is scalable, suitable for both pretraining and continued pretraining, and works well in multi-task settings. CRISP performs favorably compared to other methods that adjust the training distribution of the generalist data with guidance from the limited domain-specific data. Our findings demonstrate improvements across different domains in terms of language modeling perplexity and accuracy on multiple-choice question tasks. We also present ablation studies that examine the impact of dataset sizes, clustering configurations, and model sizes.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 50 of its 69 references that are in the map. First posted on arXiv 30 Sep 2024.

Builds on

50 of 69 references.

References · 50

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  9. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  10. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  11. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  12. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
38 more
  1. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  2. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  3. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  4. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  5. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  6. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  7. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  8. 2023Scaling Data-Constrained Language Modelscited by 403
  9. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 399
  10. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  11. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 243
  12. 2024A Survey on Data Selection for Language Modelscited by 236
  13. 2023PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Financecited by 222
  14. 2022Datamodels: Predicting Predictions from Training Datacited by 189
  15. 2020Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Modelscited by 169
  16. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 161
  17. 2020Unsupervised Domain Clusters in Pretrained Language Modelscited by 160
  18. 2019Data Valuation using Reinforcement Learningcited by 116
  19. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 100
  20. 2022A framework for bilevel optimization that enables stochastic and global variance reduction algorithmscited by 99
  21. 2023To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisiscited by 92
  22. 2016Edinburgh Neural Machine Translation Systems for WMT 16cited by 76
  23. 2023The MiniPile Challenge for Data-Efficient Language Modelscited by 75
  24. 2024Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Modelscited by 72
  25. 2017Learning to select data for transfer learning with Bayesian Optimizationcited by 64
  26. 2024Automatic Data Curation for Self-Supervised Learning: A Clustering-Based Approachcited by 50
  27. 2018Denoising Neural Machine Translation Training with Trusted Data and Online Data Selectioncited by 43
  28. 2018Dual Conditional Cross-Entropy Filtering of Noisy Parallel Corporacited by 40
  29. 2020Why resampling outperforms reweighting for correcting sampling bias with stochastic gradientscited by 34
  30. 2021Meta-Learning to Improve Pre-Trainingcited by 27
  31. 2023IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Modelscited by 27
  32. 2024Automated Data Curation for Robust Language Model Fine-Tuningcited by 27
  33. 2023Adaptive Training Distributions with Scalable Online Bilevel Optimizationcited by 11
  34. 2021On the Complementarity of Data Selection and Fine Tuning for Domain Adaptationcited by 10
  35. 2021Data Summarization via Bilevel Optimizationcited by 8
  36. 2024Cleaner Pretraining Corpus Curation with Neural Web Scrapingcited by 5
  37. 2024Large Language Model-guided Document Selectioncited by 5
  38. 2024Domain-Specific Pretraining of Language Models: A Comparative Study in the Medical Fieldcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 11; 2026: 6.

18 papers.

Sort:

By area:

  1. 2025Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-trainingcited by 66
  2. 2025Scaling Laws for Optimal Data Mixturescited by 52
  3. 2025Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generationcited by 21
  4. 2025Language Models Improve When Pretraining Data Matches Target Taskscited by 19
  5. 2025Closing the Gap Between Text and Speech Understanding in LLMscited by 17
  6. 2025Pretraining with hierarchical memories: separating long-tail and common knowledgecited by 14
  7. 2026Multi-Task GRPO: Reliable LLM Reasoning Across Taskscited by 14
  8. 2024Training Bilingual LMs with Data Constraints in the Targeted Languagecited by 6
  9. 2026Scaling Laws for Mixture Pretraining Under Data Constraintscited by 6
  10. 2025GRAPE: Optimize Data Mixture for Group Robust Multi-target Adaptive Pretrainingcited by 4
  11. 2025Soup-of-Experts: Pretraining Specialist Models via Parameters Averagingcited by 3
  12. 2025Removing Noise, not Finding Gold: Quality Filtering for Large-Scale Pretrainingcited by 3
6 more
  1. 2026Cram Less to Fit More: Training Data Pruning Improves Memorization of Factscited by 3
  2. 2025Assessing the Role of Data Quality in Training Bilingual Language Modelscited by 2
  3. 2026Optimal Splitting of Language Models from Mixtures to Specialized Domainscited by 1
  4. 2025LLM on a Budget: Active Knowledge Distillation for Efficient Classification of Large Text Corporacited by 0
  5. 2026Influence Guided Sampling for Domain Adaptation of Text Retrieverscited by 0
  6. 2026Multilingual Knowledge Transfer under Data Constraints via Lexical Interventionscited by 0