# SEA-HELM: Southeast Asian Holistic Evaluation of Language Models

arXiv: https://arxiv.org/abs/2502.14301 · HTML: https://mlsanity.com/paper/2502.14301/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 108 references that are in the map.
- First posted on arXiv 20 Feb 2025.

## Builds on (most cited first, 30 of 50)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://mlsanity.com/paper/2003.05002/) (2020)
- [Benchmarking Large Language Models for News Summarization](https://mlsanity.com/paper/2301.13848/) (2023)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation](https://mlsanity.com/paper/2302.09210/) (2023)
- [XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages](https://mlsanity.com/paper/2106.13822/) (2021)
- [XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning](https://mlsanity.com/paper/2005.00333/) (2020)
- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://mlsanity.com/paper/2308.16884/) (2023)
- [Reasoning with Language Model Prompting: A Survey](https://mlsanity.com/paper/2212.09597/) (2022)
- [Challenges and Strategies in Cross-Cultural NLP](https://mlsanity.com/paper/2203.10020/) (2022)
- [Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation](https://mlsanity.com/paper/2412.03304/) (2024)
- [Towards Measuring and Modeling "Culture" in LLMs: A Survey](https://mlsanity.com/paper/2403.15412/) (2024)
- [Power to the People? Opportunities and Challenges for Participatory AI](https://mlsanity.com/paper/2209.07572/) (2022)
- [Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier](https://mlsanity.com/paper/2412.04261/) (2024)
- [SeaLLMs -- Large Language Models for Southeast Asia](https://mlsanity.com/paper/2312.00738/) (2023)
- [SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoning](https://mlsanity.com/paper/2309.04766/) (2023)
- [The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts](https://mlsanity.com/paper/2401.13136/) (2024)
- [MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task](https://mlsanity.com/paper/2410.03983/) (2024)
- [Evaluating the Social Impact of Generative AI Systems in Systems and Society](https://mlsanity.com/paper/2306.05949/) (2023)
- [Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition](https://mlsanity.com/paper/2004.03066/) (2020)

## Built on by (most cited first, 19 of 19)

- [FilBench: Can LLMs Understand and Generate Filipino?](https://mlsanity.com/paper/2508.03523/) (2025)
- [Mangosteen: An Open Thai Corpus for Language Model Pretraining](https://mlsanity.com/paper/2507.14664/) (2025)
- [LaoBench: A Large-Scale Multidimensional Lao Benchmark for Large Language Models](https://mlsanity.com/paper/2511.11334/) (2025)
- [Buy versus Build an LLM: A Decision Framework for Governments](https://mlsanity.com/paper/2602.13033/) (2026)
- [BURMESE-SAN: Burmese NLP Benchmark for Evaluating Large Language Models](https://mlsanity.com/paper/2602.18788/) (2026)
- [MATA: Mindful Assessment of the Telugu Abilities of Large Language Models](https://mlsanity.com/paper/2508.13526/) (2025)
- [SiamGPT: Quality-First Fine-Tuning for Stable Thai Text Generation](https://mlsanity.com/paper/2512.19455/) (2025)
- [Script Sensitivity: Benchmarking Language Models on Unicode, Romanized and Mixed-Script Sinhala](https://mlsanity.com/paper/2601.14958/) (2026)
- [Phoenix-VL 1.5 Medium Technical Report](https://mlsanity.com/paper/2605.10391/) (2026)
- [The Annotation Scarcity Paradox in Low-Resource NLP Evaluation: A Decade of Acceleration and Emerging Constraints](https://mlsanity.com/paper/2605.19066/) (2026)
- [Evaluation of Chunking Strategies for Effective Text Embedding in Low-Resource Language on Agricultural Documents](https://mlsanity.com/paper/2605.22203/) (2026)
- [SEA-NLI: Natural Language Inference as a Lens into Southeast Asian Cultural Understanding](https://mlsanity.com/paper/2606.03284/) (2026)
- [Multilingual Fine-Tuning via Localized Gradient Conflict Resolution](https://mlsanity.com/paper/2606.05613/) (2026)
- [MSQA: A Natively Sourced Multilingual and Multicultural SimpleQA Benchmark](https://mlsanity.com/paper/2607.00724/) (2026)
- [From Global Benchmarks to Local Evaluations: Benchmarking LLMs for the German Public Sector](https://mlsanity.com/paper/2608.17827/) (2026)
- [CultureConverse: A Multilingual Multi-turn Simulation Harness for Culturally Grounded Assistance in East and Southeast Asia](https://mlsanity.com/paper/2608.28405/) (2026)
- [SEA-LION-v4.8: A Technical Report](https://mlsanity.com/paper/2609.18310/) (2026)
- [SEA-CLIP-Tiny: Efficient Multilingual Text-Vision Embedding for Southeast Asian Languages](https://mlsanity.com/paper/2609.30739/) (2026)
- [SEA-HELM Audio: Evaluating Multilingual ASR-LLM Pipeline Performance Across SEA-HELM Subsets](https://mlsanity.com/paper/2610.19869/) (2026)

## Abstract (arXiv)

With the rapid emergence of novel capabilities in Large Language Models (LLMs), the need for rigorous multilingual and multicultural benchmarks that are integrated has become more pronounced. Though existing LLM benchmarks are capable of evaluating specific capabilities of LLMs in English as well as in various mid- to low-resource languages, including those in the Southeast Asian (SEA) region, a comprehensive and culturally representative evaluation suite for the SEA languages has not been developed thus far. Here, we present SEA-HELM, a holistic linguistic and cultural LLM evaluation suite that emphasises SEA languages, comprising five core pillars: (1) NLP Classics, (2) LLM-specifics, (3) SEA Linguistics, (4) SEA Culture, (5) Safety. SEA-HELM currently supports Filipino, Indonesian, Tamil, Thai, and Vietnamese. We also introduce the SEA-HELM leaderboard, which allows users to understand models' multilingual and multicultural performance in a systematic and user-friendly manner. We make the SEA-HELM evaluation code publicly available.