mlsanityData to 7 Oct 2026

SeaExam and SeaBench: Benchmarking LLMs with Local Multilingual Questions in Southeast Asia

arXiv 2502.06298 ↗First posted 10 Feb 2025cs.CL cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

This study introduces two novel benchmarks, SeaExam and SeaBench, designed to evaluate the capabilities of Large Language Models (LLMs) in Southeast Asian (SEA) application scenarios. Unlike existing multilingual datasets primarily derived from English translations, these benchmarks are constructed based on real-world scenarios from SEA regions. SeaExam draws from regional educational exams to form a comprehensive dataset that encompasses subjects such as local history and literature. In contrast, SeaBench is crafted around multi-turn, open-ended tasks that reflect daily interactions within SEA communities. Our evaluations demonstrate that SeaExam and SeaBench more effectively discern LLM performance on SEA language tasks compared to their translated benchmarks. This highlights the importance of using real-world queries to assess the multilingual capabilities of LLMs.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 22 of its 31 references that are in the map. First posted on arXiv 10 Feb 2025.

Builds on

22 of 31 references.

References · 22

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  5. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  6. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  7. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  8. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  9. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  10. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  11. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  12. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
10 more
  1. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  2. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
  3. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  4. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 163
  5. 2023SeaLLMs -- Large Language Models for Southeast Asiacited by 104
  6. 2023SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoningcited by 94
  7. 2023Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?cited by 87
  8. 2024SeaLLMs 3: Open Foundation and Chat Multilingual Large Language Models for Southeast Asian Languagescited by 35
  9. 2024WildVis: Open Source Visualizer for Million-Scale Chat Logs in the Wildcited by 20
  10. 2023BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Modelscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 3.

13 papers.

Sort:

  1. 2025Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMscited by 17
  2. 2025Disentangling Language and Culture for Evaluating Multilingual Large Language Modelscited by 15
  3. 2025Hire Your Anthropologist! Rethinking Culture Benchmarks Through an Anthropological Lenscited by 13
  4. 2025SeaLLMs-Audio: Large Audio-Language Models for Southeast Asiacited by 7
  5. 2025Batayan: A Filipino NLP benchmark for evaluating Large Language Modelscited by 5
  6. 2025MELAC: Massive Evaluation of Large Language Models with Alignment of Culture in Persian Languagecited by 3
  7. 2025FilBench: Can LLMs Understand and Generate Filipino?cited by 3
  8. 2025WangchanThaiInstruct: An instruction-following Dataset for Culture-Aware, Multitask, and Multi-domain Evaluation in Thaicited by 3
  9. 2025LaoBench: A Large-Scale Multidimensional Lao Benchmark for Large Language Modelscited by 2
  10. 2026BURMESE-SAN: Burmese NLP Benchmark for Evaluating Large Language Modelscited by 1
  11. 2025MATA: Mindful Assessment of the Telugu Abilities of Large Language Modelscited by 0
  12. 2026Who Benchmarks the Benchmarks? A Case Study of LLM Evaluation in Icelandiccited by 0
1 more
  1. 2026SEATauBench: Progressively Adapting Tool-Agent-User Evaluation Into Low-Resource Southeast Asian Languagescited by 0