# WangchanBERTa: Pretraining transformer-based Thai Language Models

arXiv: https://arxiv.org/abs/2101.09635 · HTML: https://mlsanity.com/paper/2101.09635/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 41 references that are in the map.
- First posted on arXiv 24 Jan 2021.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://mlsanity.com/paper/2010.00133/) (2020)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [The Annotation Guideline of LST20 Corpus](https://mlsanity.com/paper/2008.05055/) (2020)

## Built on by (most cited first, 26 of 26)

- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [Impact of Tokenization on Language Models: An Analysis for Turkish](https://mlsanity.com/paper/2204.08832/) (2022)
- [IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation](https://mlsanity.com/paper/2104.08200/) (2021)
- [Typhoon: Thai Large Language Models](https://mlsanity.com/paper/2312.13951/) (2023)
- [PyThaiNLP: Thai Natural Language Processing in Python](https://mlsanity.com/paper/2312.04649/) (2023)
- [How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in Japanese](https://mlsanity.com/paper/2306.09572/) (2023)
- [Egalitarian Language Representation in Language Models: It All Begins with Tokenizers](https://mlsanity.com/paper/2409.11501/) (2024)
- [BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Models](https://mlsanity.com/paper/2309.06085/) (2023)
- [Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisions](https://mlsanity.com/paper/2503.03862/) (2025)
- [Exploring Tokenization Strategies and Vocabulary Sizes for Enhanced Arabic Language Models](https://mlsanity.com/paper/2403.11130/) (2024)
- [Responsible Multilingual Large Language Models: A Survey of Development, Applications, and Societal Impact](https://mlsanity.com/paper/2410.17532/) (2024)
- [WangchanThaiInstruct: An instruction-following Dataset for Culture-Aware, Multitask, and Multi-domain Evaluation in Thai](https://mlsanity.com/paper/2508.15239/) (2025)
- [PhayaThaiBERT: Enhancing a Pretrained Thai Language Model with Unassimilated Loanwords](https://mlsanity.com/paper/2311.12475/) (2023)
- [Representing the Under-Represented: Cultural and Core Capability Benchmarks for Developing Thai Large Language Models](https://mlsanity.com/paper/2410.04795/) (2024)
- [Mangosteen: An Open Thai Corpus for Language Model Pretraining](https://mlsanity.com/paper/2507.14664/) (2025)
- [Thai Semantic End-of-Turn Detection for Real-Time Voice Agents](https://mlsanity.com/paper/2510.04016/) (2025)
- [Misspelling Semantics In Thai](https://mlsanity.com/paper/2206.09680/) (2022)
- [Label-Aware Automatic Verbalizer for Few-Shot Text Classification](https://mlsanity.com/paper/2310.12778/) (2023)
- [NusaBERT: Teaching IndoBERT to be Multilingual and Multicultural](https://mlsanity.com/paper/2403.01817/) (2024)
- [Thai Universal Dependency Treebank](https://mlsanity.com/paper/2405.07586/) (2024)
- [ThaiCoref: Thai Coreference Resolution Dataset](https://mlsanity.com/paper/2406.06000/) (2024)
- [Empowering Global Voices: A Data-Efficient, Phoneme-Tone Adaptive Approach to High-Fidelity Speech Synthesis](https://mlsanity.com/paper/2504.07858/) (2025)
- [Explainable Depression Detection using Masked Hard Instance Mining](https://mlsanity.com/paper/2505.24609/) (2025)
- [Aspect-Level Obfuscated Sentiment in Thai Financial Disclosures and Its Impact on Abnormal Returns](https://mlsanity.com/paper/2511.13481/) (2025)
- [ThaiTrees: Thai Syntactic Dependency Trees Across Domains](https://mlsanity.com/paper/2609.27558/) (2026)
- [Single-Sample Decoding of a Two-Layer Sound-Word Canvas Model for Thai Klon Pad](https://mlsanity.com/paper/2610.14980/) (2026)

## Abstract (arXiv)

Transformer-based language models, more specifically BERT-based architectures have achieved state-of-the-art performance in many downstream tasks. However, for a relatively low-resource language such as Thai, the choices of models are limited to training a BERT-based model based on a much smaller dataset or finetuning multi-lingual models, both of which yield suboptimal downstream performance. Moreover, large-scale multi-lingual pretraining does not take into account language-specific features for Thai. To overcome these limitations, we pretrain a language model based on RoBERTa-base architecture on a large, deduplicated, cleaned training set (78GB in total size), curated from diverse domains of social media posts, news articles and other publicly available datasets. We apply text processing rules that are specific to Thai most importantly preserving spaces, which are important chunk and sentence boundaries in Thai before subword tokenization. We also experiment with word-level, syllable-level and SentencePiece tokenization with a smaller dataset to explore the effects on tokenization on downstream performance. Our model wangchanberta-base-att-spm-uncased trained on the 78.5GB dataset outperforms strong baselines (NBSVM, CRF and ULMFit) and multi-lingual models (XLMR and mBERT) on both sequence classification and token classification tasks in human-annotated, mono-lingual contexts.