# A standardized Project Gutenberg corpus for statistical analysis of natural language and quantitative linguistics

arXiv: https://arxiv.org/abs/1812.08092 · HTML: https://mlsanity.com/paper/1812.08092/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Dec 2018.

## Built on by (most cited first, 30 of 64)

- [Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora](https://mlsanity.com/paper/2504.08165/) (2025)
- [Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNs](https://mlsanity.com/paper/2403.19827/) (2024)
- [Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus](https://mlsanity.com/paper/2301.11796/) (2023)
- [From Tokens to Words: On the Inner Lexicon of LLMs](https://mlsanity.com/paper/2410.05864/) (2024)
- [Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora](https://mlsanity.com/paper/2412.05149/) (2024)
- [LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recovery](https://mlsanity.com/paper/2310.18356/) (2023)
- [On the State of the Art in Authorship Attribution and Authorship Verification](https://mlsanity.com/paper/2209.06869/) (2022)
- [[Call for Papers] The 2nd BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus](https://mlsanity.com/paper/2404.06214/) (2024)
- [Avoiding Copyright Infringement via Large Language Model Unlearning](https://mlsanity.com/paper/2406.10952/) (2024)
- [BabyLM Turns 3: Call for papers for the 2025 BabyLM workshop](https://mlsanity.com/paper/2502.10645/) (2025)
- [Phoneme-Level BERT for Enhanced Prosody of Text-to-Speech with Grapheme Predictions](https://mlsanity.com/paper/2301.08810/) (2023)
- [BOUQuET: dataset, Benchmark and Open initiative for Universal Quality Evaluation in Translation](https://mlsanity.com/paper/2502.04314/) (2025)
- [Not all layers are equally as important: Every Layer Counts BERT](https://mlsanity.com/paper/2311.02265/) (2023)
- [Controllable Data Generation by Deep Learning: A Review](https://mlsanity.com/paper/2207.09542/) (2022)
- [Investigating Critical Period Effects in Language Acquisition through Neural Language Models](https://mlsanity.com/paper/2407.19325/) (2024)
- [Can Pre-trained Language Models Interpret Similes as Smart as Human?](https://mlsanity.com/paper/2203.08452/) (2022)
- [PART: Pre-trained Authorship Representation Transformer](https://mlsanity.com/paper/2209.15373/) (2022)
- [BabyLM Turns 4 and Goes Multilingual: Call for Papers for the 2026 BabyLM Workshop](https://mlsanity.com/paper/2602.20092/) (2026)
- [Understanding writing style in social media with a supervised contrastively pre-trained transformer](https://mlsanity.com/paper/2310.11081/) (2023)
- [Beyond Scaling Laws: Understanding Transformer Performance with Associative Memory](https://mlsanity.com/paper/2405.08707/) (2024)
- [Once Upon a Time: Interactive Learning for Storytelling with Small Language Models](https://mlsanity.com/paper/2509.15714/) (2025)
- [AntLM: Bridging Causal and Masked Language Models](https://mlsanity.com/paper/2412.03275/) (2024)
- [BabyStories: Can Reinforcement Learning Teach Baby Language Models to Write Better Stories?](https://mlsanity.com/paper/2310.16681/) (2023)
- [Customizing Large Language Model Generation Style using Parameter-Efficient Finetuning](https://mlsanity.com/paper/2409.04574/) (2024)
- [Joint prediction of truecasing and punctuation for conversational speech in low-resource scenarios](https://mlsanity.com/paper/2109.06103/) (2021)
- [Risks of AI Foundation Models in Education](https://mlsanity.com/paper/2110.10024/) (2021)
- [Entropy and type-token ratio in gigaword corpora](https://mlsanity.com/paper/2411.10227/) (2024)
- [A Methodology for Generative Spelling Correction via Natural Spelling Errors Emulation across Multiple Domains and Languages](https://mlsanity.com/paper/2308.09435/) (2023)
- [BAMBINO-LM: (Bilingual-)Human-Inspired Continual Pretraining of BabyLM](https://mlsanity.com/paper/2406.11418/) (2024)
- [M2QA: Multi-domain Multilingual Question Answering](https://mlsanity.com/paper/2407.01091/) (2024)

## Abstract (arXiv)

The use of Project Gutenberg (PG) as a text corpus has been extremely popular in statistical analysis of language for more than 25 years. However, in contrast to other major linguistic datasets of similar importance, no consensual full version of PG exists to date. In fact, most PG studies so far either consider only a small number of manually selected books, leading to potential biased subsets, or employ vastly different pre-processing strategies (often specified in insufficient details), raising concerns regarding the reproducibility of published results. In order to address these shortcomings, here we present the Standardized Project Gutenberg Corpus (SPGC), an open science approach to a curated version of the complete PG data containing more than 50,000 books and more than 3 × 109 word-tokens. Using different sources of annotated metadata, we not only provide a broad characterization of the content of PG, but also show different examples highlighting the potential of SPGC for investigating language variability across time, subjects, and authors. We publish our methodology in detail, the code to download and process the data, as well as the obtained corpus itself on 3 different levels of granularity (raw text, timeseries of word tokens, and counts of words). In this way, we provide a reproducible, pre-processed, full-size version of Project Gutenberg as a new scientific resource for corpus linguistics, natural language processing, and information retrieval.