mlsanityData to 14 Oct 2026

The German Commons - 154 Billion Tokens of Openly Licensed Text for German Language Models

arXiv 2510.13996 ↗First posted 15 Oct 2025cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Large language model development relies on large-scale training corpora, yet most contain data of unclear licensing status, limiting the development of truly open models. This problem is exacerbated for non-English languages, where openly licensed text remains critically scarce. We introduce the German Commons, the largest collection of openly licensed German text to date. It compiles data from 41 sources across seven domains, encompassing legal, scientific, cultural, political, news, economic, and web text. Through systematic sourcing from established data providers with verifiable licensing, it yields 154.56 billion tokens of high-quality text for language model training. Our processing pipeline implements comprehensive quality filtering, deduplication, and text formatting fixes, ensuring consistent quality across heterogeneous text sources. All domain subsets feature licenses of at least CC-BY-SA 4.0 or equivalent, ensuring legal compliance for model training and redistribution. The German Commons therefore addresses the critical gap in openly licensed German pretraining data, and enables the development of truly open German language models. We also release code for corpus construction and data filtering tailored to German language text, rendering the German Commons fully reproducible and extensible.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 28 of its 54 references that are in the map. First posted on arXiv 15 Oct 2025.

Builds on

28 of 54 references.

References · 28

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  4. 2018Datasheets for Datasetscited by 1,386
  5. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  6. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  7. 2016Bag of Tricks for Efficient Text Classificationcited by 876
  8. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  9. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  10. 2016FastText.zip: Compressing text classification modelscited by 335
  11. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 182
  12. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 182
16 more
  1. 2022OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and conceptscited by 177
  2. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 174
  3. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  4. 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 143
  5. 2025olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Modelscited by 125
  6. 2021MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfercited by 90
  7. 2023The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AIcited by 73
  8. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  9. 2023Talkin' 'Bout AI Generation: Copyright and the Generative-AI Supply Chaincited by 46
  10. 2025The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Textcited by 31
  11. 2020Towards an Open Platform for Legal Informationcited by 19
  12. 2024MOSEL: 950,000 Hours of Speech Data for Open-Source Speech Foundation Model Training on EU Languagescited by 14
  13. 2025OCR Error Post-Correction with LLMs in Historical Documents: No Free Lunchescited by 11
  14. 2024Toxicity of the Commons: Curating Open-Source Pre-Training Datacited by 7
  15. 2025The KL3M Data Project: Copyright-Clean Training Resources for Large Language Modelscited by 6
  16. 2025A Common Pool of Privacy Problems: Legal and Technical Lessons from a Large-Scale Web-Scraped Machine Learning Datasetcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026propella-1: Multi-Property Document Annotation for LLM Data Curation at Scalecited by 6
  2. 2026KletterMix: Climbing Toward High-Quality German Pretraining Data - The Full Reportcited by 1
  3. 2026A Sovereign, Open-Source Foundation Model for German and Englishcited by 1