mlsanityData to 14 Oct 2026

SEA-HELM: Southeast Asian Holistic Evaluation of Language Models

arXiv 2502.14301 ↗First posted 20 Feb 2025cs.CL cs.AIcited by 19

Not on the map. Why?

Abstract (arXiv)

With the rapid emergence of novel capabilities in Large Language Models (LLMs), the need for rigorous multilingual and multicultural benchmarks that are integrated has become more pronounced. Though existing LLM benchmarks are capable of evaluating specific capabilities of LLMs in English as well as in various mid- to low-resource languages, including those in the Southeast Asian (SEA) region, a comprehensive and culturally representative evaluation suite for the SEA languages has not been developed thus far. Here, we present SEA-HELM, a holistic linguistic and cultural LLM evaluation suite that emphasises SEA languages, comprising five core pillars: (1) NLP Classics, (2) LLM-specifics, (3) SEA Linguistics, (4) SEA Culture, (5) Safety. SEA-HELM currently supports Filipino, Indonesian, Tamil, Thai, and Vietnamese. We also introduce the SEA-HELM leaderboard, which allows users to understand models' multilingual and multicultural performance in a systematic and user-friendly manner. We make the SEA-HELM evaluation code publicly available.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 50 of its 108 references that are in the map. First posted on arXiv 20 Feb 2025.

Builds on

50 of 108 references.

References · 50

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  4. 2023Qwen Technical Reportcited by 3,344
  5. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  6. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  7. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  8. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
  9. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  10. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  11. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  12. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 488
38 more
  1. 2023Benchmarking Large Language Models for News Summarizationcited by 445
  2. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 437
  3. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 348
  4. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  5. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  6. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  7. 2022Reasoning with Language Model Prompting: A Surveycited by 252
  8. 2022Challenges and Strategies in Cross-Cultural NLPcited by 218
  9. 2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 191
  10. 2024Towards Measuring and Modeling "Culture" in LLMs: A Surveycited by 167
  11. 2022Power to the People? Opportunities and Challenges for Participatory AIcited by 138
  12. 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 129
  13. 2023SeaLLMs -- Large Language Models for Southeast Asiacited by 104
  14. 2023SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoningcited by 94
  15. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contextscited by 94
  16. 2024MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Taskcited by 93
  17. 2023Evaluating the Social Impact of Generative AI Systems in Systems and Societycited by 92
  18. 2020Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESuppositioncited by 77
  19. 2024A Complete Survey on LLM-based AI Chatbotscited by 71
  20. 2023Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarizationcited by 69
  21. 2022NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languagescited by 68
  22. 2023Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyondcited by 65
  23. 2023Multi-lingual and Multi-cultural Figurative Language Understandingcited by 47
  24. 2021CLiMP: A Benchmark for Chinese Language Model Evaluationcited by 46
  25. 2019When Choosing Plausible Alternatives, Clever Hans can be Clevercited by 40
  26. 2024IndoCulture: Exploring Geographically-Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provincescited by 39
  27. 2024Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languagescited by 38
  28. 2019Translationese as a Language in "Multilingual" NMTcited by 36
  29. 2024SeaLLMs 3: Open Foundation and Chat Multilingual Large Language Models for Southeast Asian Languagescited by 36
  30. 2021A Large-scale Dataset for Hate Speech Detection on Vietnamese Social Media Textscited by 25
  31. 2021IndoNLI: A Natural Language Inference Dataset for Indonesiancited by 24
  32. 2021NOPE: A Corpus of Naturally-Occurring Presuppositions in Englishcited by 23
  33. 2024Multilingual Blending: LLM Safety Alignment Evaluation with Language Mixturecited by 22
  34. 2021Pragmatic competence of pre-trained language models through the lens of discourse connectivescited by 17
  35. 2024Benchmark Inflation: Revealing LLM Performance Gaps Using Retro-Holdoutscited by 14
  36. 2023Expectations over Unspoken Alternatives Predict Pragmatic Inferencescited by 9
  37. 2024Self-training Large Language Models through Knowledge Detectioncited by 8
  38. 2024Socially Responsible Data for Large Multilingual Language Modelscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 14.

19 papers.

Sort:

By area:

  1. 2025FilBench: Can LLMs Understand and Generate Filipino?cited by 3
  2. 2025Mangosteen: An Open Thai Corpus for Language Model Pretrainingcited by 2
  3. 2025LaoBench: A Large-Scale Multidimensional Lao Benchmark for Large Language Modelscited by 2
  4. 2026Buy versus Build an LLM: A Decision Framework for Governmentscited by 1
  5. 2026BURMESE-SAN: Burmese NLP Benchmark for Evaluating Large Language Modelscited by 1
  6. 2025MATA: Mindful Assessment of the Telugu Abilities of Large Language Modelscited by 0
  7. 2025SiamGPT: Quality-First Fine-Tuning for Stable Thai Text Generationcited by 0
  8. 2026Script Sensitivity: Benchmarking Language Models on Unicode, Romanized and Mixed-Script Sinhalacited by 0
  9. 2026Phoenix-VL 1.5 Medium Technical Reportcited by 0
  10. 2026The Annotation Scarcity Paradox in Low-Resource NLP Evaluation: A Decade of Acceleration and Emerging Constraintscited by 0
  11. 2026Evaluation of Chunking Strategies for Effective Text Embedding in Low-Resource Language on Agricultural Documentscited by 0
  12. 2026SEA-NLI: Natural Language Inference as a Lens into Southeast Asian Cultural Understandingcited by 0
7 more
  1. 2026Multilingual Fine-Tuning via Localized Gradient Conflict Resolutioncited by 0
  2. 2026MSQA: A Natively Sourced Multilingual and Multicultural SimpleQA Benchmarkcited by 0
  3. 2026From Global Benchmarks to Local Evaluations: Benchmarking LLMs for the German Public Sectorcited by 0
  4. 2026CultureConverse: A Multilingual Multi-turn Simulation Harness for Culturally Grounded Assistance in East and Southeast Asiacited by 0
  5. 2026SEA-LION-v4.8: A Technical Reportcited by 0
  6. 2026SEA-CLIP-Tiny: Efficient Multilingual Text-Vision Embedding for Southeast Asian Languagescited by 0
  7. 2026SEA-HELM Audio: Evaluating Multilingual ASR-LLM Pipeline Performance Across SEA-HELM Subsetscited by 0