FuLG: 150B Romanian Corpus for Language Model Pretraining
Not on the map. Why?
Abstract (arXiv)
Research in the field of language models is rapidly evolving, with many open models being released to the public. Openly available pretraining corpora usually focus on only a handful of languages, with many others either missing completely or extremely underrepresented. In this report, we introduce FuLG, a hundred-fifty-billion-token Romanian corpus extracted from CommonCrawl. We present our methodology for filtering FuLG and compare it via ablation studies against existing Romanian corpora.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 12 of its 28 references that are in the map. First posted on arXiv 18 Jul 2024.
Builds on
12 of 28 references.
References · 12
- 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
- 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
- 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
- 2024OLMo: Accelerating the Science of Language Modelscited by 613
- 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
- 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
- 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
- 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
- 2016FastText.zip: Compressing text classification modelscited by 335
- 2023LLM360: Towards Fully Transparent Open-Source LLMscited by 104
- 2024MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Seriescited by 67
Built on by
Citing papers per year: 2025: 1.
1 paper.