# Typhoon: Thai Large Language Models

arXiv: https://arxiv.org/abs/2312.13951 · HTML: https://mlsanity.com/paper/2312.13951/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 58 references that are in the map.
- First posted on arXiv 21 Dec 2023.

## Builds on (most cited first, 30 of 43)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://mlsanity.com/paper/2308.08747/) (2023)
- [RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback](https://mlsanity.com/paper/2309.00267/) (2023)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages](https://mlsanity.com/paper/2106.13822/) (2021)
- [XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning](https://mlsanity.com/paper/2005.00333/) (2020)
- [MEGA: Multilingual Evaluation of Generative AI](https://mlsanity.com/paper/2303.12528/) (2023)

## Built on by (most cited first, 30 of 30)

- [Survey of Cultural Awareness in Language Models: Text and Beyond](https://mlsanity.com/paper/2411.00860/) (2024)
- [On the Societal Impact of Open Foundation Models](https://mlsanity.com/paper/2403.07918/) (2024)
- [CultureLLM: Incorporating Cultural Differences into Large Language Models](https://mlsanity.com/paper/2402.10946/) (2024)
- [CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries](https://mlsanity.com/paper/2501.01282/) (2025)
- [CulturePark: Boosting Cross-cultural Understanding in Large Language Models](https://mlsanity.com/paper/2405.15145/) (2024)
- [The Model Openness Framework: Promoting Completeness and Openness for Reproducibility, Transparency, and Usability in Artificial Intelligence](https://mlsanity.com/paper/2403.13784/) (2024)
- [How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of Target Language Text?](https://mlsanity.com/paper/2406.11477/) (2024)
- [Sailor: Open Language Models for South-East Asia](https://mlsanity.com/paper/2404.03608/) (2024)
- [The AI Community Building the Future? A Quantitative Analysis of Development Activity on Hugging Face Hub](https://mlsanity.com/paper/2405.13058/) (2024)
- [Typhoon 2: A Family of Open Text and Multimodal Thai Large Language Models](https://mlsanity.com/paper/2412.13702/) (2024)
- [Sabi'a-2: A New Generation of Portuguese Large Language Models](https://mlsanity.com/paper/2403.09887/) (2024)
- [The Future of Open Human Feedback](https://mlsanity.com/paper/2408.16961/) (2024)
- [Enhancing Low-Resource Language and Instruction Following Capabilities of Audio Language Models](https://mlsanity.com/paper/2409.10999/) (2024)
- [OpenThaiGPT 1.5: A Thai-Centric Open Source Large Language Model](https://mlsanity.com/paper/2411.07238/) (2024)
- [Building High-Quality Datasets for Portuguese LLMs: From Common Crawl Snapshots to Industrial-Grade Corpora](https://mlsanity.com/paper/2509.08824/) (2025)
- [Adapting Chat Language Models Using Only Target Unlabeled Language Data](https://mlsanity.com/paper/2412.11704/) (2024)
- [CAReDiO: Cultural Alignment via Representativeness and Distinctiveness Guided Data Optimization](https://mlsanity.com/paper/2504.08820/) (2025)
- [Typhoon T1: An Open Thai Reasoning Model](https://mlsanity.com/paper/2502.09042/) (2025)
- [OpenThaiGPT 1.6 and R1: Thai-Centric Open Source and Reasoning Large Language Models](https://mlsanity.com/paper/2504.01789/) (2025)
- [Thai Winograd Schemas: A Benchmark for Thai Commonsense Reasoning](https://mlsanity.com/paper/2405.18375/) (2024)
- [Eir: Thai Medical Large Language Models](https://mlsanity.com/paper/2409.08523/) (2024)
- [Vocabulary Customization for Efficient Domain-Specific LLM Deployment](https://mlsanity.com/paper/2509.26124/) (2025)
- [Bailong: Bilingual Transfer Learning based on QLoRA and Zip-tie Embedding](https://mlsanity.com/paper/2404.00862/) (2024)
- [GECKO: Generative Language Model for English, Code and Korean](https://mlsanity.com/paper/2405.15640/) (2024)
- [JAI-1: A Thai-Centric Large Language Model](https://mlsanity.com/paper/2510.08620/) (2025)
- [Fluent Alignment with Disfluent Judges: Post-training for Lower-resource Languages](https://mlsanity.com/paper/2512.08777/) (2025)
- [TSWAP: A Multilingual Retrieval-Augmented Thai Wellness Advisor](https://mlsanity.com/paper/2608.22917/) (2026)
- [CultureConverse: A Multilingual Multi-turn Simulation Harness for Culturally Grounded Assistance in East and Southeast Asia](https://mlsanity.com/paper/2608.28405/) (2026)
- [SEA-LION-v4.8: A Technical Report](https://mlsanity.com/paper/2609.18310/) (2026)
- [Large-scale factor analysis shows machine intelligence is only partially interpretable](https://mlsanity.com/paper/2609.36515/) (2026)

## Abstract (arXiv)

Typhoon is a series of Thai large language models (LLMs) developed specifically for the Thai language. This technical report presents challenges and insights in developing Thai LLMs, including data preparation, pretraining, instruction-tuning, and evaluation. As one of the challenges of low-resource languages is the amount of pretraining data, we apply continual training to transfer existing world knowledge from a strong LLM. To evaluate the Thai knowledge encapsulated in each model from the pretraining stage, we develop ThaiExam, a benchmark based on examinations for high-school students and investment professionals in Thailand. In addition, we fine-tune Typhoon to follow Thai instructions, and we evaluate instruction-tuned models on Thai instruction datasets as well as translation, summarization, and question-answering tasks. Experimental results on a suite of Thai benchmarks show that Typhoon outperforms all open-source Thai language models, and its performance is on par with GPT-3.5 in Thai while having only 7 billion parameters and being 2.62 times more efficient in tokenizing Thai text.