# Dynaword: From One-shot to Continuously Developed Datasets

arXiv: https://arxiv.org/abs/2508.02271 · HTML: https://mlsanity.com/paper/2508.02271/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 37 references that are in the map.
- First posted on arXiv 4 Aug 2025.

## Builds on (most cited first, 17 of 17)

- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023)
- [Towards a Cleaner Document-Oriented Multilingual Crawled Corpus](https://mlsanity.com/paper/2201.06642/) (2022)
- [MMTEB: Massive Multilingual Text Embedding Benchmark](https://mlsanity.com/paper/2502.13595/) (2025)
- [A Note on Shumailov et al. (2024): `AI Models Collapse When Trained on Recursively Generated Data'](https://mlsanity.com/paper/2410.12954/) (2024)
- [Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Tasks](https://mlsanity.com/paper/2406.13469/) (2024)
- [The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embedding](https://mlsanity.com/paper/2406.02396/) (2024)
- [Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training](https://mlsanity.com/paper/2506.01732/) (2025)
- [The Danish Gigaword Project](https://mlsanity.com/paper/2005.03521/) (2020)
- [Towards Best Practices for Open Datasets for LLM Training](https://mlsanity.com/paper/2501.08365/) (2025)
- [Danish Foundation Models](https://mlsanity.com/paper/2311.07264/) (2023)
- [Corpora Compared: The Case of the Swedish Gigaword & Wikipedia Corpora](https://mlsanity.com/paper/2011.03281/) (2020)

## Built on by (most cited first, 3 of 3)

- [SDUs DAISY: A Benchmark for Danish Culture](https://mlsanity.com/paper/2601.19930/) (2026)
- [DFM Mimir v1: An Open HRM Delivering Frontier Performance at 1B Parameters Using Only Permissible Post-Training Data](https://mlsanity.com/paper/2608.13517/) (2026)
- [Multilingual GSM-Symbolic: What determines capability transfer across languages?](https://mlsanity.com/paper/2610.03367/) (2026)

## Abstract (arXiv)

Large-scale datasets are foundational for research and development in natural language processing. However, current approaches face three key challenges: (1) reliance on ambiguously licensed sources restricting use, sharing, and derivative works; (2) static dataset releases that prevent community contributions and diminish longevity; and (3) quality assurance processes restricted to publishing teams rather than leveraging community expertise. To address these limitations, we introduce two contributions: the Dynaword approach and Danish Dynaword. The Dynaword approach is a framework for creating large-scale, open datasets that can be continuously updated through community collaboration. Danish Dynaword is a concrete implementation that validates this approach and demonstrates its potential. Danish Dynaword contains over four times as many tokens as comparable releases, is exclusively openly licensed, and has received multiple contributions across industry and research. The repository includes light-weight tests to ensure data formatting, quality, and documentation, establishing a sustainable framework for ongoing community contributions and dataset evolution.