# SeaExam and SeaBench: Benchmarking LLMs with Local Multilingual Questions in Southeast Asia

arXiv: https://arxiv.org/abs/2502.06298 · HTML: https://mlsanity.com/paper/2502.06298/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 31 references that are in the map.
- First posted on arXiv 10 Feb 2025.

## Builds on (most cited first, 22 of 22)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://mlsanity.com/paper/2406.11939/) (2024)
- [MEGA: Multilingual Evaluation of Generative AI](https://mlsanity.com/paper/2303.12528/) (2023)
- [WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild](https://mlsanity.com/paper/2406.04770/) (2024)
- [ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning](https://mlsanity.com/paper/2304.05613/) (2023)
- [Benchmarking Foundation Models with Language-Model-as-an-Examiner](https://mlsanity.com/paper/2306.04181/) (2023)
- [Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting](https://mlsanity.com/paper/2305.07004/) (2023)
- [SeaLLMs -- Large Language Models for Southeast Asia](https://mlsanity.com/paper/2312.00738/) (2023)
- [SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoning](https://mlsanity.com/paper/2309.04766/) (2023)
- [Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?](https://mlsanity.com/paper/2309.07462/) (2023)
- [SeaLLMs 3: Open Foundation and Chat Multilingual Large Language Models for Southeast Asian Languages](https://mlsanity.com/paper/2407.19672/) (2024)
- [WildVis: Open Source Visualizer for Million-Scale Chat Logs in the Wild](https://mlsanity.com/paper/2409.03753/) (2024)
- [BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Models](https://mlsanity.com/paper/2309.06085/) (2023)

## Built on by (most cited first, 13 of 13)

- [Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMs](https://mlsanity.com/paper/2502.12982/) (2025)
- [Disentangling Language and Culture for Evaluating Multilingual Large Language Models](https://mlsanity.com/paper/2505.24635/) (2025)
- [Hire Your Anthropologist! Rethinking Culture Benchmarks Through an Anthropological Lens](https://mlsanity.com/paper/2510.05931/) (2025)
- [SeaLLMs-Audio: Large Audio-Language Models for Southeast Asia](https://mlsanity.com/paper/2511.01670/) (2025)
- [Batayan: A Filipino NLP benchmark for evaluating Large Language Models](https://mlsanity.com/paper/2502.14911/) (2025)
- [MELAC: Massive Evaluation of Large Language Models with Alignment of Culture in Persian Language](https://mlsanity.com/paper/2508.00673/) (2025)
- [FilBench: Can LLMs Understand and Generate Filipino?](https://mlsanity.com/paper/2508.03523/) (2025)
- [WangchanThaiInstruct: An instruction-following Dataset for Culture-Aware, Multitask, and Multi-domain Evaluation in Thai](https://mlsanity.com/paper/2508.15239/) (2025)
- [LaoBench: A Large-Scale Multidimensional Lao Benchmark for Large Language Models](https://mlsanity.com/paper/2511.11334/) (2025)
- [BURMESE-SAN: Burmese NLP Benchmark for Evaluating Large Language Models](https://mlsanity.com/paper/2602.18788/) (2026)
- [MATA: Mindful Assessment of the Telugu Abilities of Large Language Models](https://mlsanity.com/paper/2508.13526/) (2025)
- [Who Benchmarks the Benchmarks? A Case Study of LLM Evaluation in Icelandic](https://mlsanity.com/paper/2603.16406/) (2026)
- [SEATauBench: Progressively Adapting Tool-Agent-User Evaluation Into Low-Resource Southeast Asian Languages](https://mlsanity.com/paper/2606.28715/) (2026)

## Abstract (arXiv)

This study introduces two novel benchmarks, SeaExam and SeaBench, designed to evaluate the capabilities of Large Language Models (LLMs) in Southeast Asian (SEA) application scenarios. Unlike existing multilingual datasets primarily derived from English translations, these benchmarks are constructed based on real-world scenarios from SEA regions. SeaExam draws from regional educational exams to form a comprehensive dataset that encompasses subjects such as local history and literature. In contrast, SeaBench is crafted around multi-turn, open-ended tasks that reflect daily interactions within SEA communities. Our evaluations demonstrate that SeaExam and SeaBench more effectively discern LLM performance on SEA language tasks compared to their translated benchmarks. This highlights the importance of using real-world queries to assess the multilingual capabilities of LLMs.