# Advancing the Evaluation of Traditional Chinese Language Models: Towards a Comprehensive Benchmark Suite

arXiv: https://arxiv.org/abs/2309.08448 · HTML: https://mlsanity.com/paper/2309.08448/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 19 references that are in the map.
- First posted on arXiv 15 Sep 2023.

## Builds on (most cited first, 12 of 12)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Models](https://mlsanity.com/paper/2305.13711/) (2023)
- [DRCD: a Chinese Machine Reading Comprehension Dataset](https://mlsanity.com/paper/1806.00920/) (2018)
- [Extending the Pre-Training of BLOOM for Improved Support of Traditional Chinese: Models, Methods and Results](https://mlsanity.com/paper/2303.04715/) (2023)

## Built on by (most cited first, 14 of 14)

- [Taiwan LLM: Bridging the Linguistic Divide with a Culturally Aligned Language Model](https://mlsanity.com/paper/2311.17487/) (2023)
- [An Improved Traditional Chinese Evaluation Suite for Foundation Model](https://mlsanity.com/paper/2403.01858/) (2024)
- [Characterizing Bias: Benchmarking Large Language Models in Simplified versus Traditional Chinese](https://mlsanity.com/paper/2505.22645/) (2025)
- [InstructionCP: A fast approach to transfer Large Language Models into target language](https://mlsanity.com/paper/2405.20175/) (2024)
- [Breeze-7B Technical Report](https://mlsanity.com/paper/2403.02712/) (2024)
- [The Breeze 2 Herd of Models: Traditional Chinese LLMs Based on Llama with Vision-Aware and Function-Calling Capabilities](https://mlsanity.com/paper/2501.13921/) (2025)
- [VisTW: Benchmarking Vision-Language Models for Traditional Chinese in Taiwan](https://mlsanity.com/paper/2503.10427/) (2025)
- [Measuring Taiwanese Mandarin Language Understanding](https://mlsanity.com/paper/2403.20180/) (2024)
- [Knowledge Retrieval Based on Generative AI](https://mlsanity.com/paper/2501.04635/) (2025)
- [Measuring Hong Kong Massive Multi-Task Language Understanding](https://mlsanity.com/paper/2505.02177/) (2025)
- [Hyacinth6B: A large language model for Traditional Chinese](https://mlsanity.com/paper/2403.13334/) (2024)
- [Bailong: Bilingual Transfer Learning based on QLoRA and Zip-tie Embedding](https://mlsanity.com/paper/2404.00862/) (2024)
- [Multi-TW: Benchmarking Multimodal Models on Traditional Chinese Question Answering in Taiwan](https://mlsanity.com/paper/2508.01274/) (2025)
- [OpenTWBench (Open Taiwan Benchmark): Taiwan-Domain Knowledge Evaluation for Large Language Models, and How the Measurement Apparatus Shapes the Score](https://mlsanity.com/paper/2610.19848/) (2026)

## Abstract (arXiv)

The evaluation of large language models is an essential task in the field of language understanding and generation. As language models continue to advance, the need for effective benchmarks to assess their performance has become imperative. In the context of Traditional Chinese, there is a scarcity of comprehensive and diverse benchmarks to evaluate the capabilities of language models, despite the existence of certain benchmarks such as DRCD, TTQA, CMDQA, and FGC dataset. To address this gap, we propose a novel set of benchmarks that leverage existing English datasets and are tailored to evaluate language models in Traditional Chinese. These benchmarks encompass a wide range of tasks, including contextual question-answering, summarization, classification, and table understanding. The proposed benchmarks offer a comprehensive evaluation framework, enabling the assessment of language models' capabilities across different tasks. In this paper, we evaluate the performance of GPT-3.5, Taiwan-LLaMa-v1.0, and Model 7-C, our proprietary model, on these benchmarks. The evaluation results highlight that our model, Model 7-C, achieves performance comparable to GPT-3.5 with respect to a part of the evaluated capabilities. In an effort to advance the evaluation of language models in Traditional Chinese and stimulate further research in this field, we have open-sourced our benchmark and opened the model for trial.