SWEb: A Large Web Dataset for the Scandinavian Languages
Not on the map. Why?
Abstract (arXiv)
This paper presents the hitherto largest pretraining dataset for the Scandinavian languages: the Scandinavian WEb (SWEb), comprising over one trillion tokens. The paper details the collection and processing pipeline, and introduces a novel model-based text extractor that significantly reduces complexity in comparison with rule-based approaches. We also introduce a new cloze-style benchmark for evaluating language models in Swedish, and use this test to compare models trained on the SWEb data to models trained on FineWeb, with competitive results. All data, models and code are shared openly.
Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Builds on 13 of its 24 references that are in the map. First posted on arXiv 6 Oct 2024.
Builds on
13 of 24 references.
References · 13
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
- 2020Longformer: The Long-Document Transformercited by 3,393
- 2018Deep contextualized word representationscited by 2,286
- 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,815
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,126
- 2016Bag of Tricks for Efficient Text Classificationcited by 872
- 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 760
- 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
- 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 477
- 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
- 2024A New Massive Multilingual Dataset for High-Performance Language Technologiescited by 56
- 2023ScandEval: A Benchmark for Scandinavian Natural Language Processingcited by 16
1 more
Built on by
Citing papers per year: 2026: 1.
1 paper.