The Nordic Pile: A 1.2TB Nordic Dataset for Language Modeling
Not on the map. Why?
Abstract (arXiv)
Pre-training Large Language Models (LLMs) require massive amounts of text data, and the performance of the LLMs typically correlates with the scale and quality of the datasets. This means that it may be challenging to build LLMs for smaller languages such as Nordic ones, where the availability of text corpora is limited. In order to facilitate the development of the LLMS in the Nordic languages, we curate a high-quality dataset consisting of 1.2TB of text, in all of the major North Germanic languages (Danish, Icelandic, Norwegian, and Swedish), as well as some high-quality English data. This paper details our considerations and processes for collecting, cleaning, and filtering the dataset.
Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 11 of its 21 references that are in the map. First posted on arXiv 30 Mar 2023.
Builds on
11 of 21 references.
References · 11
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
- 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
- 2016Bag of Tricks for Efficient Text Classificationcited by 869
- 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
- 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
- 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
- 2020The Pushshift Reddit Datasetcited by 289
- 2020A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languagescited by 131
- 2021Scaling Laws for Neural Machine Translationcited by 110
Built on by
Citing papers per year: 2023: 3; 2024: 0; 2025: 1.
4 papers.