# Taiwan LLM: Bridging the Linguistic Divide with a Culturally Aligned Language Model

arXiv: https://arxiv.org/abs/2311.17487 · HTML: https://mlsanity.com/paper/2311.17487/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 12 of its 22 references that are in the map.
- First posted on arXiv 29 Nov 2023.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 12 of 12)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance](https://mlsanity.com/paper/2305.17306/) (2023)
- [Advancing the Evaluation of Traditional Chinese Language Models: Towards a Comprehensive Benchmark Suite](https://mlsanity.com/paper/2309.08448/) (2023)

## Built on by (most cited first, 29 of 29)

- [Survey of Cultural Awareness in Language Models: Text and Beyond](https://mlsanity.com/paper/2411.00860/) (2024)
- [All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages](https://mlsanity.com/paper/2411.16508/) (2024)
- [CultureLLM: Incorporating Cultural Differences into Large Language Models](https://mlsanity.com/paper/2402.10946/) (2024)
- [CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries](https://mlsanity.com/paper/2501.01282/) (2025)
- [CulturePark: Boosting Cross-cultural Understanding in Large Language Models](https://mlsanity.com/paper/2405.15145/) (2024)
- [PersianLLaMA: Towards Building First Persian Large Language Model](https://mlsanity.com/paper/2312.15713/) (2023)
- [An Improved Traditional Chinese Evaluation Suite for Foundation Model](https://mlsanity.com/paper/2403.01858/) (2024)
- [Sabi'a-2: A New Generation of Portuguese Large Language Models](https://mlsanity.com/paper/2403.09887/) (2024)
- [CRAFT: Extracting and Tuning Cultural Instructions from the Wild](https://mlsanity.com/paper/2405.03138/) (2024)
- [Characterizing Bias: Benchmarking Large Language Models in Simplified versus Traditional Chinese](https://mlsanity.com/paper/2505.22645/) (2025)
- [Analyzing Nobel Prize Literature with Large Language Models](https://mlsanity.com/paper/2410.18142/) (2024)
- [The Breeze 2 Herd of Models: Traditional Chinese LLMs Based on Llama with Vision-Aware and Function-Calling Capabilities](https://mlsanity.com/paper/2501.13921/) (2025)
- [Can LLMs Grasp Implicit Cultural Values? Benchmarking LLMs' Cultural Intelligence with CQ-Bench](https://mlsanity.com/paper/2504.01127/) (2025)
- [Continual Pre-Training is (not) What You Need in Domain Adaption](https://mlsanity.com/paper/2504.13603/) (2025)
- [Measuring Taiwanese Mandarin Language Understanding](https://mlsanity.com/paper/2403.20180/) (2024)
- [Exploring Cross-lingual Latent Transplantation: Mutual Opportunities and Open Challenges](https://mlsanity.com/paper/2412.12686/) (2024)
- [Knowledge Retrieval Based on Generative AI](https://mlsanity.com/paper/2501.04635/) (2025)
- [Do Large Language Models Know Folktales? A Case Study of Yokai in Japanese Folktales](https://mlsanity.com/paper/2506.03619/) (2025)
- [Measuring Hong Kong Massive Multi-Task Language Understanding](https://mlsanity.com/paper/2505.02177/) (2025)
- [Towards culturally-appropriate conversational AI for health in the majority world: An exploratory study with citizens and professionals in Latin America](https://mlsanity.com/paper/2507.01719/) (2025)
- [TWGuard: A Case Study of LLM Safety Guardrails for Localized Linguistic Contexts](https://mlsanity.com/paper/2604.16542/) (2026)
- [TW-LegalBench: Measuring Taiwanese Legal Understanding](https://mlsanity.com/paper/2606.18699/) (2026)
- [Hyacinth6B: A large language model for Traditional Chinese](https://mlsanity.com/paper/2403.13334/) (2024)
- [Bailong: Bilingual Transfer Learning based on QLoRA and Zip-tie Embedding](https://mlsanity.com/paper/2404.00862/) (2024)
- [GPT-4's One-Dimensional Mapping of Morality: How the Accuracy of Country-Estimates Depends on Moral Domain](https://mlsanity.com/paper/2407.16886/) (2024)
- [Efficient Training of Robust Traditional Chinese LLaMA-1B on a Single Consumer GPU: Continual Pre-training, SFT, and DPO](https://mlsanity.com/paper/2510.01616/) (2025)
- [Taiwan Safety Benchmark and Breeze Guard: Toward Trustworthy AI for Taiwanese Mandarin](https://mlsanity.com/paper/2603.07286/) (2026)
- [BlueMagpie-TTS: A Token-Efficient Tokenizer, Language Model, and TTS for Taiwanese-Accent Code-Switching Speech](https://mlsanity.com/paper/2607.06054/) (2026)
- [CantoneseLLM v2: Reasoning in a Low-Resource Language](https://mlsanity.com/paper/2609.06970/) (2026)

## Abstract (arXiv)

In the realm of language models, the nuanced linguistic and cultural intricacies of Traditional Chinese, as spoken in Taiwan, have been largely overlooked. This paper introduces Taiwan LLM, a pioneering Large Language Model that specifically caters to the Traditional Chinese language, with a focus on the variant used in Taiwan. Leveraging a comprehensive pretraining corpus and instruction-finetuning datasets, we have developed a model that not only understands the complexities of Traditional Chinese but also embodies the cultural context of Taiwan. Taiwan LLM represents the first of its kind, a model that is not only linguistically accurate but also culturally resonant with its user base. Our evaluations demonstrate that Taiwan LLM achieves superior performance in understanding and generating Traditional Chinese text, outperforming existing models that are predominantly trained on Simplified Chinese or English. The open-source release of Taiwan LLM invites collaboration and further innovation, ensuring that the linguistic diversity of Chinese speakers is embraced and well-served. The model, datasets, and further resources are made publicly available to foster ongoing research and development in this field.