mlsanityData to 7 Oct 2026

The Pile: An 800GB Dataset of Diverse Text for Language Modeling

arXiv 2101.00027 ↗First posted 31 Dec 2020cs.CLcited by 2,346

Not on the map. Why?

Cited by 118 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent work has demonstrated that increased training dataset diversity improves general cross-domain knowledge and downstream generalization capability for large-scale language models. With this in mind, we present textitthe Pile: an 825 GiB English text corpus targeted at training large-scale language models. The Pile is constructed from 22 diverse high-quality subsets -- both existing and newly constructed -- many of which derive from academic or professional sources. Our evaluation of the untuned performance of GPT-2 and GPT-3 on the Pile shows that these models struggle on many of its components, such as academic writing. Conversely, models trained on the Pile improve significantly over both Raw CC and CC-100 on all components of the Pile, while improving performance on downstream evaluations. Through an in-depth exploratory analysis, we document potentially concerning aspects of the data for prospective users. We make publicly available the code used in its construction.

Key facts. Cited by 2,346 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 39; 2022: 176; 2023: 500; 2024: 714; 2025: 544; 2026: 373.

2,346 papers.

Sort:

By area:

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  11. 2023A Survey of Large Language Modelscited by 2,537
  12. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
38 more
  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  3. 2024Mixtral of Expertscited by 1,785
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  5. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  6. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  8. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  9. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  10. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  11. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  12. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  13. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  14. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  15. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  16. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  17. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  18. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  19. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  20. 2023BloombergGPT: A Large Language Model for Financecited by 923
  21. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  22. 2021Measuring Coding Challenge Competence With APPScited by 879
  23. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  24. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  25. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  26. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  27. 2023StarCoder: may the source be with you!cited by 790
  28. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  29. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  30. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  31. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  32. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  33. 2024DeepSeek-V3 Technical Reportcited by 724
  34. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  35. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  36. 2024Chronos: Learning the Language of Time Seriescited by 712
  37. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  38. 2021Deduplicating Training Data Makes Language Models Bettercited by 705

50 most cited of 2,346 All 2,346 (JSON)