mlsanityData to 7 Oct 2026

The MiniPile Challenge for Data-Efficient Language Models

arXiv 2304.08442 ↗First posted 17 Apr 2023cs.CL cs.LGcited by 74

Not on the map. Why?

Abstract (arXiv)

The ever-growing diversity of pre-training text corpora has equipped language models with generalization capabilities across various downstream tasks. However, such diverse datasets are often too large for academic budgets; hence, most research on Transformer architectures, training procedures, optimizers, etc. gets conducted on smaller, homogeneous datasets. To this end, we present The MiniPile Challenge, where one pre-trains a language model on a diverse text corpus containing at most 1M documents. MiniPile is a 6GB subset of the deduplicated 825GB The Pile corpus. To curate MiniPile, we perform a simple, three-step data filtering process: we (1) infer embeddings for all documents of the Pile, (2) cluster the embedding space using k-means, and (3) filter out low-quality clusters. To verify MiniPile's suitability for language model pre-training, we use it to pre-train a BERT and T5 model, yielding a performance drop of only 1.9%/2.5% on the GLUE and SNI benchmarks compared to the original pre-trained checkpoints trained on 2.6x/745x the amount of data. MiniPile is available at https://huggingface.co/datasets/JeanKaddour/minipile.

Key facts. Cited by 74 papers in the map (data through 7 Oct 2026). Builds on 50 of its 61 references that are in the map. First posted on arXiv 17 Apr 2023.

Builds on

50 of 61 references.

References · 50

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2023A Survey of Large Language Modelscited by 2,537
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  11. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  12. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
38 more
  1. 2020GLU Variants Improve Transformercited by 1,724
  2. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  3. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  4. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  5. 2021Ethical and social risks of harm from Language Modelscited by 957
  6. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  7. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  8. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  9. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  10. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  11. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  12. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  13. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  14. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  15. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
  16. 2020Data and its (dis)contents: A survey of dataset development and use in machine learning researchcited by 259
  17. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  18. 2017Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Ratescited by 196
  19. 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 147
  20. 2021Large-Scale Differentially Private BERTcited by 117
  21. 2022Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Datasetcited by 113
  22. 2022Causal Machine Learning: A Survey and Open Problemscited by 107
  23. 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 106
  24. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97
  25. 2023SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networkscited by 82
  26. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  27. 2021How to Train BERT with an Academic Budgetcited by 80
  28. 2022When Do Flat Minima Optimizers Work?cited by 78
  29. 2023Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpuscited by 62
  30. 2022Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averagingcited by 54
  31. 2022Language Modelling with Pixelscited by 53
  32. 2021Addressing "Documentation Debt" in Machine Learning Research: A Retrospective Datasheet for BookCorpuscited by 46
  33. 2023Spawrious: A Benchmark for Fine Control of Spurious Correlation Biasescited by 41
  34. 2022Nonparametric Masked Language Modelingcited by 39
  35. 2021NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Frameworkcited by 34
  36. 2021Benchmarking Differential Privacy and Federated Learning for BERT Modelscited by 28
  37. 2020Probabilistic Active Meta-Learningcited by 23
  38. 2020Pitfalls in Machine Learning Research: Reexamining the Development Cyclecited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 26; 2025: 30; 2026: 11.

74 papers.

Sort:

By area:

  1. 2023Challenges and Applications of Large Language Modelscited by 251
  2. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 180
  3. 2024Are We Done with MMLU?cited by 178
  4. 2024Tokenization Is More Than Compressioncited by 87
  5. 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 86
  6. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  7. 2023D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruningcited by 68
  8. 2024MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Seriescited by 67
  9. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Modelscited by 58
  10. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  11. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  12. 2024Memory3: Language Modeling with Explicit Memorycited by 39
38 more
  1. 2024Greed is All You Need: An Evaluation of Tokenizer Inference Methodscited by 34
  2. 2025Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barriercited by 31
  3. 2025The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Trainingcited by 27
  4. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routingcited by 26
  5. 2024Analysing The Impact of Sequence Composition on Language Model Pre-Trainingcited by 22
  6. 2024Balanced Data Sampling for Language Model Training with Clusteringcited by 22
  7. 2025Evolution Strategies at the Hyperscalecited by 22
  8. 2024StableMask: Refining Causal Masking in Decoder-only Transformercited by 20
  9. 2025LiMuon: Light and Fast Muon Optimizer for Large Modelscited by 20
  10. 2024Setting the Record Straight on Transformer Oversmoothingcited by 19
  11. 2024Task-Adaptive Pretrained Language Models via Clustered-Importance Samplingcited by 18
  12. 2023Data Management For Training Large Language Models: A Surveycited by 17
  13. 2024GoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill and Extreme KV-Cache Compressioncited by 17
  14. 2024A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inferencecited by 16
  15. 2025Causal Estimation of Tokenisation Biascited by 16
  16. 2024Improving Generalization and Convergence by Enhancing Implicit Regularizationcited by 14
  17. 2025Tokenization is Sensitive to Language Variationcited by 12
  18. 2025Bayesian Influence Functions for Hessian-Free Data Attributioncited by 12
  19. 2025VOCABTRIM: Vocabulary Pruning for Efficient Speculative Decoding in LLMscited by 11
  20. 2024Retrieval is Accurate Generationcited by 10
  21. 2024LLMGuard: Guarding Against Unsafe LLM Behaviorcited by 10
  22. 2025Repetitions are not all alike: distinct mechanisms sustain repetition in language modelscited by 9
  23. 2024A Review of the Challenges with Massive Web-mined Corpora Used in Large Language Models Pre-Trainingcited by 7
  24. 2025Decomposing Representation Space into Interpretable Subspaces with Unsupervised Learningcited by 7
  25. 2025RWKV-X: A Linear Complexity Hybrid Language Modelcited by 6
  26. 2026From Atoms to Trees: Building a Structured Feature Forest with Hierarchical Sparse Autoencoderscited by 6
  27. 2025KV-Latent: Dimensional-level KV Cache Reduction with Frequency-aware Rotary Positional Embeddingcited by 5
  28. 2024Is Training Data Quality or Quantity More Impactful to Small Language Model Performance?cited by 4
  29. 2025FlexiGPT: Pruning and Extending Large Language Models with Low-Rank Weight Sharingcited by 4
  30. 2023ChapGTP, ILLC's Attempt at Raising a BabyLM: Improving Data Efficiency by Automatic Task Formationcited by 3
  31. 2025CharBench: Evaluating the Role of Tokenization in Character-Level Taskscited by 3
  32. 2025Probabilistic Token Alignment for Large Language Model Fusioncited by 3
  33. 2025Noise-Adaptive Layerwise Learning Rates: Accelerating Geometry-Aware Optimization for Deep Neural Network Trainingcited by 3
  34. 2025Transformers as Intrinsic Optimizers: Forward Inference through the Energy Principlecited by 3
  35. 2024Accelerating Large Language Model Pretraining via LFR Pedagogy: Learn, Focus, and Reviewcited by 2
  36. 2025Practical Principles for AI Cost and Compute Accountingcited by 2
  37. 2025Pre-Training Curriculum for Multi-Token Prediction in Language Modelscited by 2
  38. 2025Dataset Ownership Verification for Pre-trained Masked Modelscited by 2

50 most cited of 74 All 74 (JSON)