# Representing the Under-Represented: Cultural and Core Capability Benchmarks for Developing Thai Large Language Models

arXiv: https://arxiv.org/abs/2410.04795 · HTML: https://mlsanity.com/paper/2410.04795/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 37 references that are in the map.
- First posted on arXiv 7 Oct 2024.

## Builds on (most cited first, 22 of 22)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Evaluating Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2310.19736/) (2023)
- [SeaLLMs -- Large Language Models for Southeast Asia](https://mlsanity.com/paper/2312.00738/) (2023)
- [Extrapolating Large Language Models to Non-English by Aligning Languages](https://mlsanity.com/paper/2308.04948/) (2023)
- [PolyLM: An Open Source Polyglot Large Language Model](https://mlsanity.com/paper/2307.06018/) (2023)
- [SeaLLMs 3: Open Foundation and Chat Multilingual Large Language Models for Southeast Asian Languages](https://mlsanity.com/paper/2407.19672/) (2024)
- [A Survey of Useful LLM Evaluation](https://mlsanity.com/paper/2406.00936/) (2024)
- [PyThaiNLP: Thai Natural Language Processing in Python](https://mlsanity.com/paper/2312.04649/) (2023)
- [WangchanBERTa: Pretraining transformer-based Thai Language Models](https://mlsanity.com/paper/2101.09635/) (2021)
- [Open Ko-LLM Leaderboard: Evaluating Large Language Models in Korean with Ko-H5 Benchmark](https://mlsanity.com/paper/2405.20574/) (2024)
- [SQuARe: A Large-Scale Dataset of Sensitive Questions and Acceptable Responses Created Through Human-Machine Collaboration](https://mlsanity.com/paper/2305.17696/) (2023)
- [scb-mt-en-th-2020: A Large English-Thai Parallel Corpus](https://mlsanity.com/paper/2007.03541/) (2020)
- [A Challenging Benchmark for Low-Resource Learning](https://mlsanity.com/paper/2303.03840/) (2023)

## Built on by (most cited first, 2 of 2)

- [Assessing Thai Dialect Performance in LLMs with Automatic Benchmarks and Human Evaluation](https://mlsanity.com/paper/2504.05898/) (2025)
- [TAU: A Benchmark for Cultural Sound Understanding Beyond Semantics](https://mlsanity.com/paper/2509.26329/) (2025)

## Abstract (arXiv)

The rapid advancement of large language models (LLMs) has highlighted the need for robust evaluation frameworks that assess their core capabilities, such as reasoning, knowledge, and commonsense, leading to the inception of certain widely-used benchmark suites such as the H6 benchmark. However, these benchmark suites are primarily built for the English language, and there exists a lack thereof for under-represented languages, in terms of LLM development, such as Thai. On the other hand, developing LLMs for Thai should also include enhancing the cultural understanding as well as core capabilities. To address these dual challenge in Thai LLM research, we propose two key benchmarks: Thai-H6 and Thai Cultural and Linguistic Intelligence Benchmark (ThaiCLI). Through a thorough evaluation of various LLMs with multi-lingual capabilities, we provide a comprehensive analysis of the proposed benchmarks and how they contribute to Thai LLM development. Furthermore, we will make both the datasets and evaluation code publicly available to encourage further research and development for Thai LLMs.