Dynaword: From One-shot to Continuously Developed Datasets
Not on the map. Why?
Abstract (arXiv)
Large-scale datasets are foundational for research and development in natural language processing. However, current approaches face three key challenges: (1) reliance on ambiguously licensed sources restricting use, sharing, and derivative works; (2) static dataset releases that prevent community contributions and diminish longevity; and (3) quality assurance processes restricted to publishing teams rather than leveraging community expertise. To address these limitations, we introduce two contributions: the Dynaword approach and Danish Dynaword. The Dynaword approach is a framework for creating large-scale, open datasets that can be continuously updated through community collaboration. Danish Dynaword is a concrete implementation that validates this approach and demonstrates its potential. Danish Dynaword contains over four times as many tokens as comparable releases, is exclusively openly licensed, and has received multiple contributions across industry and research. The repository includes light-weight tests to ensure data formatting, quality, and documentation, establishing a sustainable framework for ongoing community contributions and dataset evolution.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 17 of its 37 references that are in the map. First posted on arXiv 4 Aug 2025.
Builds on
17 of 37 references.
References · 17
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
- 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
- 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
- 2018Datasheets for Datasetscited by 1,371
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
- 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
- 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
- 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
- 2025MMTEB: Massive Multilingual Text Embedding Benchmarkcited by 120
- 2024A Note on Shumailov et al. (2024): `AI Models Collapse When Trained on Recursively Generated Data'cited by 114
- 2024Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Taskscited by 33
- 2024The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embeddingcited by 26
5 more
- 2025Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Trainingcited by 19
- 2020The Danish Gigaword Projectcited by 17
- 2025Towards Best Practices for Open Datasets for LLM Trainingcited by 5
- 2023Danish Foundation Modelscited by 4
- 2020Corpora Compared: The Case of the Swedish Gigaword & Wikipedia Corporacited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.