# Assessing Thai Dialect Performance in LLMs with Automatic Benchmarks and Human Evaluation

arXiv: https://arxiv.org/abs/2504.05898 · HTML: https://mlsanity.com/paper/2504.05898/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 22 references that are in the map.
- First posted on arXiv 8 Apr 2025.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages](https://mlsanity.com/paper/2106.13822/) (2021)
- [InFoBench: Evaluating Instruction Following Ability in Large Language Models](https://mlsanity.com/paper/2401.03601/) (2024)
- [NusaCrowd: Open Source Initiative for Indonesian NLP Resources](https://mlsanity.com/paper/2212.09648/) (2022)
- [Filipino Benchmarks for Measuring Sexist and Homophobic Bias in Multilingual Language Models from Southeast Asia](https://mlsanity.com/paper/2412.07303/) (2024)
- [Kalahi: A handcrafted, grassroots cultural LLM evaluation suite for Filipino](https://mlsanity.com/paper/2409.15380/) (2024)
- [Seed-Free Synthetic Data Generation Framework for Instruction-Tuning LLMs: A Case Study in Thai](https://mlsanity.com/paper/2411.15484/) (2024)
- [Representing the Under-Represented: Cultural and Core Capability Benchmarks for Developing Thai Large Language Models](https://mlsanity.com/paper/2410.04795/) (2024)

## Built on by (most cited first, 5 of 5)

- [JAI-1: A Thai-Centric Large Language Model](https://mlsanity.com/paper/2510.08620/) (2025)
- [The Hrunting of AI: Where and How to Improve English Dialectal Fairness](https://mlsanity.com/paper/2603.15187/) (2026)
- [Evaluating Japanese Dialect Robustness Across Speech and Text-based Large Language Models](https://mlsanity.com/paper/2606.25436/) (2026)
- [Large-scale factor analysis shows machine intelligence is only partially interpretable](https://mlsanity.com/paper/2609.36515/) (2026)
- [ASCRIBE: Atomic and Significance-Based Reasoning for Thai Clinical SOAP Note Generation](https://mlsanity.com/paper/2610.01234/) (2026)

## Abstract (arXiv)

Large language models show promising results in various NLP tasks. Despite these successes, the robustness and consistency of LLMs in underrepresented languages remain largely unexplored, especially concerning local dialects. Existing benchmarks also focus on main dialects, neglecting LLMs' ability on local dialect texts. In this paper, we introduce a Thai local dialect benchmark covering Northern (Lanna), Northeastern (Isan), and Southern (Dambro) Thai, evaluating LLMs on five NLP tasks: summarization, question answering, translation, conversation, and food-related tasks. Furthermore, we propose a human evaluation guideline and metric for Thai local dialects to assess generation fluency and dialect-specific accuracy. Results show that LLM performance declines significantly in local Thai dialects compared to standard Thai, with only proprietary models like GPT-4o and Gemini2 demonstrating some fluency