mlsanityData to 7 Oct 2026

How to Train Data-Efficient LLMs

arXiv 2402.09668 ↗First posted 15 Feb 2024cs.LG cs.AI cs.CLcited by 91

Not on the map. Why?

Abstract (arXiv)

The training of large language models (LLMs) is expensive. In this paper, we study data-efficient approaches for pre-training LLMs, i.e., techniques that aim to optimize the Pareto frontier of model quality and training resource/data consumption. We seek to understand the tradeoffs associated with data selection routines based on (i) expensive-to-compute data-quality estimates, and (ii) maximization of coverage and diversity-based measures in the feature space. Our first technique, Ask-LLM, leverages the zero-shot reasoning capabilities of instruction-tuned LLMs to directly assess the quality of a training example. To target coverage, we propose Density sampling, which models the data distribution to select a diverse sample. In our comparison of 19 samplers, involving hundreds of evaluation tasks and pre-training runs, we find that Ask-LLM and Density are the best methods in their respective categories. Coverage sampling can recover the performance of the full data, while models trained on Ask-LLM data consistently outperform full-data training -- even when we reject 90% of the original dataset, while converging up to 70% faster.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). Builds on 57 of its 85 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

57 of 85 references.

References · 57

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
38 more
  1. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  2. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  3. 2015Teaching Machines to Read and Comprehendcited by 1,039
  4. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  5. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  6. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  7. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  8. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  9. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  10. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  11. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  12. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  13. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 492
  14. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  15. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  16. 2023Scaling Data-Constrained Language Modelscited by 391
  17. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
  18. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 337
  19. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 323
  20. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  21. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 314
  22. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  23. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
  24. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 264
  25. 2023Self-Consuming Generative Models Go MADcited by 239
  26. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
  27. 2020Coresets via Bilevel Optimization for Continual Learning and Streamingcited by 191
  28. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 180
  29. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 156
  30. 2022DeepCore: A Comprehensive Library for Coreset Selection in Deep Learningcited by 146
  31. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 146
  32. 2018Turning Big data into tiny data: Constant-size coresets for k-means, PCA and projective clusteringcited by 94
  33. 2019Accelerating Deep Learning by Focusing on the Biggest Loserscited by 94
  34. 2023Data Distillation: A Surveycited by 86
  35. 2023Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loopcited by 75
  36. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Modelscited by 65
  37. 2016Coresets and Sketchescited by 63
  38. 2023Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runscited by 52

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 33; 2025: 43; 2026: 15.

91 papers.

Sort:

By area:

  1. 2025Gemma 3 Technical Reportcited by 1,686
  2. 2024An Introduction to Vision-Language Modelingcited by 175
  3. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 172
  4. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 96
  5. 2024Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solvingcited by 82
  6. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 73
  7. 2025Ming-Omni: A Unified Multimodal Model for Perception and Generationcited by 62
  8. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Modelscited by 56
  9. 2024Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating Worldcited by 49
  10. 2024Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scalecited by 37
  11. 2025Ultra-FineWeb: Efficient Data Filtering and Verification for High-Quality LLM Training Datacited by 35
  12. 2024Harnessing Diversity for Important Data Selection in Pretraining Large Language Modelscited by 34
38 more
  1. 2025Large-Scale Data Selection for Instruction Tuningcited by 34
  2. 2025Enhancing Multilingual LLM Pretraining with Model-Based Data Selectioncited by 33
  3. 2024Data curation via joint example selection further accelerates multimodal learningcited by 32
  4. 2025Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Modelscited by 31
  5. 2025Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learningcited by 25
  6. 2025Kanana: Compute-efficient Bilingual Language Modelscited by 19
  7. 2025From Selection to Generation: A Survey of LLM-based Active Learningcited by 18
  8. 2025Language Models Improve When Pretraining Data Matches Target Taskscited by 18
  9. 2024A Systematic Assessment of OpenAI o1-Preview for Higher Order Thinking in Educationcited by 16
  10. 2025QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM Pretrainingcited by 16
  11. 2024MeanCache: User-Centric Semantic Caching for LLM Web Servicescited by 14
  12. 2024Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMscited by 14
  13. 2025FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gaincited by 14
  14. 2025Utility-Diversity Aware Online Batch Selection for LLM Supervised Fine-tuningcited by 14
  15. 2025Truly Self-Improving Agents Require Intrinsic Metacognitive Learningcited by 13
  16. 2024Selection of LLM Fine-Tuning Data based on Orthogonal Rulescited by 12
  17. 2024MDCure: A Scalable Pipeline for Multi-Document Instruction-Followingcited by 10
  18. 2025SampleMix: A Sample-wise Pre-training Data Mixing Strategey by Coordinating Data Quality and Diversitycited by 10
  19. 2024Federated Data-Efficient Instruction Tuning for Large Language Modelscited by 9
  20. 2025Comparative Analysis Based on DeepSeek, ChatGPT, and Google Gemini: Features, Techniques, Performance, Future Prospectscited by 9
  21. 2025SCIZOR: A Self-Supervised Approach to Data Curation for Large-Scale Imitation Learningcited by 9
  22. 2025PiKE: Adaptive Data Mixing for Large-Scale Multi-Task Learning Under Low Gradient Conflictscited by 8
  23. 2024Influential Language Data Selection via Gradient Trajectory Pursuitcited by 7
  24. 2024A Survey of Transformer-based Language Models with Focus on Efficiencycited by 6
  25. 2024Language Model-Driven Data Pruning Enables Efficient Active Learningcited by 6
  26. 2024Training Bilingual LMs with Data Constraints in the Targeted Languagecited by 6
  27. 2025CritiQ: Mining Data Quality Criteria from Human Preferencescited by 6
  28. 2025ALinFiK: Learning to Approximate Linearized Future Influence Kernel for Scalable Third-Party LLM Data Valuationcited by 6
  29. 2025MobileLLM-R1: Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipescited by 6
  30. 2024A Moral Imperative: The Need for Continual Superalignment of Large Language Modelscited by 5
  31. 2024Large Language Model-guided Document Selectioncited by 5
  32. 2024Automated Filtering of Human Feedback Data for Aligning Text-to-Image Diffusion Modelscited by 5
  33. 20247B Fully Open Source Moxin-LLM/VLM -- From Pretraining to GRPO-based Reinforcement Learning Enhancementcited by 5
  34. 2025FRAME: Boosting LLMs with A Four-Quadrant Multi-Stage Pretraining Strategycited by 4
  35. 2025MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretrainingcited by 4
  36. 2025Disentangling the Roles of Representation and Selection in Data Pruningcited by 4
  37. 2025Benchmarking and Improving LLM Robustness for Personalized Generationcited by 4
  38. 2024A Bayesian Approach to Data Point Selectioncited by 3

50 most cited of 91 All 91 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LEAD: Iterative Data Selection for Efficient LLM Instruction Tuning
  2. 2024Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Models
  3. 2024A Survey on Data Synthesis and Augmentation for Large Language Models
  4. 2026How Should LLMs Consume High-Quality Data? Optimal Data Scheduling via Quality-Aware Functional Scaling Laws
  5. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls
  6. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models
  7. 2026Better as Generators Than Classifiers: Leveraging LLMs and Synthetic Data for Low-Resource Multilingual Classification
  8. 2024Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling