mlsanityData to 7 Oct 2026

Datasheet for the Pile

arXiv 2201.07311 ↗First posted 13 Jan 2022cs.CLcited by 41

Not on the map. Why?

Abstract (arXiv)

This datasheet describes the Pile, a 825 GiB dataset of human-authored text compiled by EleutherAI for use in large-scale language modeling. The Pile is comprised of 22 different text sources, ranging from original scrapes done for this project, to text data made available by the data owners, to third-party scrapes available online.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 32 of its 85 references that are in the map. First posted on arXiv 13 Jan 2022.

Builds on

32 of 85 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  6. 2020Rethinking Attention with Performerscited by 1,584
  7. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  8. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  9. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  10. 2016Generative Adversarial Text to Image Synthesiscited by 735
  11. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  12. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
20 more
  1. 2020Recipes for building an open-domain chatbotcited by 593
  2. 2021Fast Model Editing at Scalecited by 491
  3. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  4. 2015Skip-Thought Vectorscited by 321
  5. 2015The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systemscited by 172
  6. 2021Addressing "Documentation Debt" in Machine Learning Research: A Retrospective Datasheet for BookCorpuscited by 46
  7. 2021Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Explorationcited by 44
  8. 2015It's a Man's Wikipedia? Assessing Gender Inequality in an Online Encyclopediacited by 30
  9. 2020Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledgecited by 24
  10. 2021Intersectional Bias in Causal Language Modelscited by 24
  11. 2021Invariant Language Modelingcited by 15
  12. 2021Cut the CARP: Fishing for zero-shot story evaluationcited by 14
  13. 2021Neural Program Generation Modulo Static Analysiscited by 13
  14. 2015First Women, Second Sex: Gender Bias in Wikipediacited by 11
  15. 2020Distill, Adapt, Distill: Training Small, In-Domain Models for Neural Machine Translationcited by 7
  16. 2018Paraphrase Detection on Noisy Subtitles in Six Languagescited by 2
  17. 2019Human-like machine thinking: Language guided imaginationcited by 2
  18. 2020Joint translation and unit conversion for end-to-end localizationcited by 2
  19. 2019Simulating Problem Difficulty in Arithmetic Cognition Through Dynamic Connectionist Modelscited by 1
  20. 2020Tilde at WMT 2020: News Task Systemscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 16; 2024: 8; 2025: 5; 2026: 8.

41 papers.

Sort:

By area:

  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  3. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  4. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  5. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  6. 2023Challenges and Applications of Large Language Modelscited by 251
  7. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  8. 2023Emergent and Predictable Memorization in Large Language Modelscited by 147
  9. 2023Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Modelscited by 144
  10. 2023Foundation Models and Fair Usecited by 112
  11. 2023HuatuoGPT-II, One-stage Training for Medical Adaption of LLMscited by 91
  12. 2023The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AIcited by 73
29 more
  1. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  2. 2023ZeRO++: Extremely Efficient Collective Communication for Giant Model Trainingcited by 50
  3. 2024Consent in Crisis: The Rapid Decline of the AI Data Commonscited by 45
  4. 2022Data Governance in the Age of Large-Scale Data-Driven Language Technologycited by 32
  5. 2024Block Transformer: Global-to-Local Language Modeling for Fast Inferencecited by 32
  6. 2022Documenting Geographically and Contextually Diverse Data Sources: The BigScience Catalogue of Language Data and Resourcescited by 11
  7. 2023Large Content And Behavior Models To Understand, Simulate, And Optimize Content And Behaviorcited by 9
  8. 2025The Lucie-7B LLM and the Lucie Training Dataset: Open resources for multilingual language generationcited by 9
  9. 2023GAIA Search: Hugging Face and Pyserini Interoperability for NLP Training Data Explorationcited by 8
  10. 2025Enhancing LLM Watermark Resilience Against Both Scrubbing and Spoofing Attackscited by 8
  11. 2024LMD3: Language Model Data Density Dependencecited by 7
  12. 2023A Language Model of Java Methods with Train/Test Deduplicationcited by 6
  13. 2024Reawakening knowledge: Anticipatory recovery from catastrophic interference via structured trainingcited by 6
  14. 2024Introducing L2M3, A Multilingual Medical Large Language Model to Advance Health Equity in Low-Resource Regionscited by 5
  15. 2024Layer Importance and Hallucination Analysis in Large Language Models via Enhanced Activation Variance-Sparsitycited by 4
  16. 2023RecycleGPT: An Autoregressive Language Model with Recyclable Modulecited by 3
  17. 2025Membership Inference Attacks on Tokenizers of Large Language Modelscited by 3
  18. 2024Teaching Human Behavior Improves Content Understanding Abilities Of LLMscited by 2
  19. 2025Probability Signature: Bridging Data Semantics and Embedding Structure in Language Modelscited by 2
  20. 2026Language Models Represent and Transform Concepts with Shared Geometrycited by 2
  21. 2025Relative Overfitting and Accept-Reject Frameworkcited by 1
  22. 2026Scaling Attention via Feature Sparsitycited by 1
  23. 2023DYAD: A Descriptive Yet Abjuring Density efficient approximation to linear neural network layerscited by 0
  24. 2026Membership Inference Attacks for Retrieval Based In-Context Learning for Document Question Answeringcited by 0
  25. 2026Adaptive Inverted-Index Routing for Granular Mixtures-of-Expertscited by 0
  26. 2026Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probingcited by 0
  27. 2026Dual Dimensionality for Local and Global Attentioncited by 0
  28. 2026The Announcement Carries the Cue: Markup, Boundaries, and the Notation of Pre-Training Corporacited by 0
  29. 2026Syntax vs. Semantics: How Transformers Learn Deep Dependenciescited by 0