mlsanityData to 7 Oct 2026

A standardized Project Gutenberg corpus for statistical analysis of natural language and quantitative linguistics

arXiv 1812.08092 ↗First posted 19 Dec 2018cs.CL cs.DL cs.IRcited by 64

Not on the map. Why?

Abstract (arXiv)

The use of Project Gutenberg (PG) as a text corpus has been extremely popular in statistical analysis of language for more than 25 years. However, in contrast to other major linguistic datasets of similar importance, no consensual full version of PG exists to date. In fact, most PG studies so far either consider only a small number of manually selected books, leading to potential biased subsets, or employ vastly different pre-processing strategies (often specified in insufficient details), raising concerns regarding the reproducibility of published results. In order to address these shortcomings, here we present the Standardized Project Gutenberg Corpus (SPGC), an open science approach to a curated version of the complete PG data containing more than 50,000 books and more than 3 × 109 word-tokens. Using different sources of annotated metadata, we not only provide a broad characterization of the content of PG, but also show different examples highlighting the potential of SPGC for investigating language variability across time, subjects, and authors. We publish our methodology in detail, the code to download and process the data, as well as the obtained corpus itself on 3 different levels of granularity (raw text, timeseries of word tokens, and counts of words). In this way, we provide a reproducible, pre-processed, full-size version of Project Gutenberg as a new scientific resource for corpus linguistics, natural language processing, and information retrieval.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 6; 2023: 10; 2024: 20; 2025: 15; 2026: 10.

64 papers.

Sort:

By area:

  1. 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 86
  2. 2024Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNscited by 65
  3. 2023Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpuscited by 62
  4. 2024Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 48
  5. 2024From Tokens to Words: On the Inner Lexicon of LLMscited by 46
  6. 2023LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recoverycited by 37
  7. 2022On the State of the Art in Authorship Attribution and Authorship Verificationcited by 27
  8. 2024[Call for Papers] The 2nd BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpuscited by 26
  9. 2025BabyLM Turns 3: Call for papers for the 2025 BabyLM workshopcited by 24
  10. 2023Phoneme-Level BERT for Enhanced Prosody of Text-to-Speech with Grapheme Predictionscited by 22
  11. 2024Avoiding Copyright Infringement via Large Language Model Unlearningcited by 22
  12. 2023Not all layers are equally as important: Every Layer Counts BERTcited by 21
38 more
  1. 2025BOUQuET: dataset, Benchmark and Open initiative for Universal Quality Evaluation in Translationcited by 21
  2. 2022Controllable Data Generation by Deep Learning: A Reviewcited by 20
  3. 2024Investigating Critical Period Effects in Language Acquisition through Neural Language Modelscited by 15
  4. 2022Can Pre-trained Language Models Interpret Similes as Smart as Human?cited by 13
  5. 2022PART: Pre-trained Authorship Representation Transformercited by 13
  6. 2024Beyond Scaling Laws: Understanding Transformer Performance with Associative Memorycited by 12
  7. 2023Understanding writing style in social media with a supervised contrastively pre-trained transformercited by 11
  8. 2025Once Upon a Time: Interactive Learning for Storytelling with Small Language Modelscited by 11
  9. 2026BabyLM Turns 4 and Goes Multilingual: Call for Papers for the 2026 BabyLM Workshopcited by 10
  10. 2024AntLM: Bridging Causal and Masked Language Modelscited by 8
  11. 2023BabyStories: Can Reinforcement Learning Teach Baby Language Models to Write Better Stories?cited by 7
  12. 2024Customizing Large Language Model Generation Style using Parameter-Efficient Finetuningcited by 7
  13. 2021Risks of AI Foundation Models in Educationcited by 6
  14. 2024Entropy and type-token ratio in gigaword corporacited by 6
  15. 2021Joint prediction of truecasing and punctuation for conversational speech in low-resource scenarioscited by 5
  16. 2023A Methodology for Generative Spelling Correction via Natural Spelling Errors Emulation across Multiple Domains and Languagescited by 5
  17. 2024BAMBINO-LM: (Bilingual-)Human-Inspired Continual Pretraining of BabyLMcited by 5
  18. 2024M2QA: Multi-domain Multilingual Question Answeringcited by 5
  19. 2022Text characterization based on recurrence networkscited by 4
  20. 2024YuLan: An Open-source Large Language Modelcited by 4
  21. 2024What Should Baby Models Read? Exploring Sample-Efficient Data Composition on Model Performancecited by 4
  22. 2025A Distributional Perspective on Word Learning in Neural Language Modelscited by 4
  23. 2023ChapGTP, ILLC's Attempt at Raising a BabyLM: Improving Data Efficiency by Automatic Task Formationcited by 3
  24. 2024From Intentions to Techniques: A Comprehensive Taxonomy and Challenges in Text Watermarking for Large Language Modelscited by 3
  25. 2025LongTail-Swap: benchmarking language models' abilities on rare wordscited by 3
  26. 2022A decomposition of book structure through ousiometric fluctuations in cumulative word-timecited by 2
  27. 2024BERTtime Stories: Investigating the Role of Synthetic Story Data in Language Pre-trainingcited by 2
  28. 2024Building, Reusing, and Generalizing Abstract Representations from Concrete Sequencescited by 2
  29. 2025Is a Peeled Apple Still Red? Evaluating LLMs' Ability for Conceptual Combination with Property Typecited by 2
  30. 2025Influence-driven Curriculum Learning for Pre-training on Limited Datacited by 2
  31. 2025Looking to Learn: Token-wise Dynamic Gating for Low-Resource Vision-Language Modellingcited by 2
  32. 2023Quantifying the Dissimilarity of Textscited by 1
  33. 2025When Personalization Tricks Detectors: The Feature-Inversion Trap in Machine-Generated Text Detectioncited by 1
  34. 2025One-shot Style Transfer LLM log-probabilities for Authorship Attribution and Verificationcited by 1
  35. 2025Sample-Efficient Language Modeling with Linear Attention and Lightweight Enhancementscited by 1
  36. 2026Computational Representations of Character Significance in Novelscited by 1
  37. 2026AuthorMix: Modular Authorship Style Transfer via Layer-wise Adapter Mixingcited by 1
  38. 2026Agent-Driven Corpus Linguistics: A Framework for Autonomous Linguistic Discoverycited by 1

50 most cited of 64 All 64 (JSON)