mlsanityData to 7 Oct 2026

SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoning

arXiv 2309.04766 ↗First posted 9 Sep 2023cs.CL cs.AIcited by 94

Not on the map. Why?

Abstract (arXiv)

We present SeaEval, a benchmark for multilingual foundation models. In addition to characterizing how these models understand and reason with natural language, we also investigate how well they comprehend cultural practices, nuances, and values. Alongside standard accuracy metrics, we investigate the brittleness of foundation models in the dimensions of semantics and multilinguality. Our analyses span both open-sourced and closed models, leading to empirical results across classic NLP tasks, reasoning, and cultural comprehension. Key findings indicate (1) Most models exhibit varied behavior when given paraphrased instructions. (2) Many models still suffer from exposure bias (e.g., positional bias, majority label bias). (3) For questions rooted in factual, scientific, and commonsense knowledge, consistent responses are expected across multilingual queries that are semantically equivalent. Yet, most models surprisingly demonstrate inconsistent performance on these queries. (4) Multilingually-trained models have not attained "balanced multilingual" capabilities. Our endeavors underscore the need for more generalizable semantic representations and enhanced multilingual contextualization. SeaEval can serve as a launchpad for more thorough investigations and evaluations for multilingual and multicultural scenarios.

Key facts. Cited by 94 papers in the map (data through 7 Oct 2026). Builds on 53 of its 105 references that are in the map. First posted on arXiv 9 Sep 2023.

Builds on

53 of 105 references.

References · 53

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  11. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
38 more
  1. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  5. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  8. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  9. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  10. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  11. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  12. 2018Neural Network Acceptability Judgmentscited by 952
  13. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  14. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  15. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  16. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  17. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  18. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  19. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 302
  20. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  21. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  22. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  23. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
  24. 2021DialogSum: A Real-Life Scenario Dialogue Summarization Datasetcited by 181
  25. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  26. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 163
  27. 2019DREAM: A Challenge Dataset and Models for Dialogue-Based Reading Comprehensioncited by 156
  28. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 156
  29. 2023From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thoughtcited by 118
  30. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 109
  31. 2020IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understandingcited by 108
  32. 2023Mitigating Label Biases for In-context Learningcited by 88
  33. 2020OCNLI: Original Chinese Natural Language Inferencecited by 86
  34. 2019Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehensioncited by 80
  35. 2023Extrapolating Large Language Models to Non-English by Aligning Languagescited by 69
  36. 2023Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluationcited by 52
  37. 2023Language Versatilists vs. Specialists: An Empirical Revisiting on Multilingual Transfer Abilitycited by 27
  38. 2020Cross-lingual Alignment Methods for Multilingual BERT: A Comparative Studycited by 24

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 43; 2025: 35; 2026: 10.

94 papers.

Sort:

By area:

  1. 2024Towards Measuring and Modeling "Culture" in LLMs: A Surveycited by 166
  2. 2024AudioBench: A Universal Benchmark for Audio Large Language Modelscited by 124
  3. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmarkcited by 118
  4. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  5. 2024Learning Planning-based Reasoning by Trajectories Collection and Process Reward Synthesizingcited by 49
  6. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  7. 2024Ethical Reasoning and Moral Value Alignment of LLMs Depend on the Language we Prompt them incited by 45
  8. 2024SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languagescited by 41
  9. 2025Culture is Not Trivia: Sociocultural Theory for Cultural NLPcited by 40
  10. 2023Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayingscited by 32
  11. 2023On Context Utilization in Summarization with Large Language Modelscited by 30
  12. 2024Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowlycited by 27
38 more
  1. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  2. 2024Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Artcited by 26
  3. 2024Code-Switching Curriculum Learning for Multilingual Transfer in LLMscited by 26
  4. 2024Extrinsic Evaluation of Cultural Competence in Large Language Modelscited by 24
  5. 2024Bridging the Language Gaps in Large Language Models with Inference-Time Cross-Lingual Interventioncited by 24
  6. 2024BertaQA: How Much Do Language Models Know About Local Culture?cited by 23
  7. 2025BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Modelscited by 23
  8. 2024Cultural Conditioning or Placebo? On the Effectiveness of Socio-Demographic Promptingcited by 22
  9. 2024Resilience of Large Language Models for Noisy Instructionscited by 20
  10. 2024PreAlign: Boosting Cross-Lingual Transfer by Early Establishment of Multilingual Alignmentcited by 20
  11. 2025Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMscited by 17
  12. 2025SEA-HELM: Southeast Asian Holistic Evaluation of Language Modelscited by 17
  13. 2025SEA-LION: Southeast Asian Languages in One Networkcited by 17
  14. 2024CIVICS: Building a Dataset for Examining Culturally-Informed Values in Large Language Modelscited by 16
  15. 2024MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routingcited by 16
  16. 2024JaColBERTv2.5: Optimising Multi-Vector Retrievers to Create State-of-the-Art Japanese Retrievers with Constrained Resourcescited by 15
  17. 2024Benchmarking Chinese Commonsense Reasoning of LLMs: From Chinese-Specifics to Reasoning-Memorization Correlationscited by 14
  18. 2024Teaching LLMs to Abstain across Languages via Multilingual Feedbackcited by 14
  19. 2025AdaMCoT: Rethinking Cross-Lingual Factual Reasoning through Adaptive Multilingual Chain-of-Thoughtcited by 14
  20. 2025CulFiT: A Fine-grained Cultural-aware LLM Training Paradigm via Multilingual Critique Data Synthesiscited by 14
  21. 2025SeaExam and SeaBench: Benchmarking LLMs with Local Multilingual Questions in Southeast Asiacited by 13
  22. 2025DRISHTIKON: A Multimodal Multilingual Benchmark for Testing Language Models' Understanding on Indian Culturecited by 13
  23. 2025CultureScope: A Dimensional Lens for Probing Cultural Understanding in LLMscited by 11
  24. 2024CrossIn: An Efficient Instruction Tuning Approach for Cross-Lingual Knowledge Alignmentcited by 10
  25. 2025CALM: Unleashing the Cross-Lingual Self-Aligning Ability of Language Model Question Answeringcited by 10
  26. 2024Do Moral Judgment and Reasoning Capability of LLMs Change with Language? A Study using the Multilingual Defining Issues Testcited by 9
  27. 2024Does Cross-Cultural Alignment Change the Commonsense Morality of Language Models?cited by 9
  28. 2024In2Core: Leveraging Influence Functions for Coreset Selection in Instruction Finetuning of Large Language Modelscited by 9
  29. 2025CCL-XCoT: An Efficient Cross-Lingual Knowledge Transfer Method for Mitigating Hallucination Generationcited by 9
  30. 2024CRAFT: Extracting and Tuning Cultural Instructions from the Wildcited by 8
  31. 2024Social Debiasing for Fair Multi-modal LLMscited by 8
  32. 2025TUMLU: A Unified and Native Language Understanding Benchmark for Turkic Languagescited by 8
  33. 2025Where Culture Fades: Revealing the Cultural Gap in Text-to-Image Generationcited by 8
  34. 2024M2Lingual: Enhancing Multilingual, Multi-Turn Instruction Alignment in Large Language Modelscited by 7
  35. 2025Analyzing Political Bias in LLMs via Target-Oriented Sentiment Classificationcited by 7
  36. 2025Cetvel: A Unified Benchmark for Evaluating Language Understanding, Generation and Cultural Capacity of LLMs for Turkishcited by 7
  37. 2024Large Language Models are Limited in Out-of-Context Knowledge Reasoningcited by 6
  38. 2024Lens: Rethinking Multilingual Enhancement for Large Language Modelscited by 5

50 most cited of 94 All 94 (JSON)