Mangosteen: An Open Thai Corpus for Language Model Pretraining
Not on the map. Why?
Abstract (arXiv)
Pre-training data shapes a language model's quality, but raw web text is noisy and demands careful cleaning. Existing large-scale corpora rely on English-centric or language-agnostic pipelines whose heuristics do not capture Thai script or cultural nuances, leaving risky material such as gambling content untreated. Prior Thai-specific efforts customize pipelines or build new ones, yet seldom release their data or document design choices, hindering reproducibility and raising the question of how to construct a transparent, high-quality Thai corpus. We introduce Mangosteen: a 47 billion-token Thai corpus built through a Thai-adapted Dolma pipeline that includes custom rule-based language ID, revised C4/Gopher quality filters, and Thai-trained content filters, plus curated non-web sources such as Wikipedia, Royal Gazette texts, OCR-extracted books, and CC-licensed YouTube subtitles. Systematic ablations using GPT-2 show the pipeline trims CommonCrawl from 202M to 25M documents while raising SEA-HELM NLG from 3 to 11; an 8B-parameter SEA-LION model continually pre-trained on Mangosteen then surpasses SEA-LION-v3 and Llama-3.1 by about four points on Thai benchmarks. We release the full pipeline code, cleaning manifests, corpus snapshot, and all checkpoints, providing a fully reproducible foundation for future Thai and regional LLM research.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 24 of its 31 references that are in the map. First posted on arXiv 19 Jul 2025.
Builds on
24 of 31 references.
References · 24
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
- 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
- 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
- 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
- 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
- 2016Bag of Tricks for Efficient Text Classificationcited by 876
- 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
- 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
12 more
- 2018Learning Word Vectors for 157 Languagescited by 280
- 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 174
- 2020A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languagescited by 131
- 2024IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languagescited by 52
- 2023FinGPT: Large Generative Models for a Small Languagecited by 49
- 2024SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languagescited by 43
- 2024Latxa: An Open Language Model and Evaluation Suite for Basquecited by 42
- 2023PyThaiNLP: Thai Natural Language Processing in Pythoncited by 28
- 2021WangchanBERTa: Pretraining transformer-based Thai Language Modelscited by 26
- 2025SEA-HELM: Southeast Asian Holistic Evaluation of Language Modelscited by 19
- 2024SambaLingo: Teaching Large Language Models New Languagescited by 11
- 2024Translation Errors Significantly Impact Low-Resource Languages in Cross-Lingual Learningcited by 9
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.