mlsanityData to 7 Oct 2026

BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models

arXiv 2502.07346 ↗First posted 11 Feb 2025cs.CLcited by 23

Not on the map. Why?

Abstract (arXiv)

Previous multilingual benchmarks focus primarily on simple understanding tasks, but for large language models(LLMs), we emphasize proficiency in instruction following, reasoning, long context understanding, code generation, and so on. However, measuring these advanced capabilities across languages is underexplored. To address the disparity, we introduce BenchMAX, a multi-way multilingual evaluation benchmark that allows for fair comparisons of these important abilities across languages. To maintain high quality, three distinct native-speaking annotators independently annotate each sample within all tasks after the data was machine-translated from English into 16 other languages. Additionally, we present a novel translation challenge stemming from dataset construction. Extensive experiments on BenchMAX reveal varying effectiveness of core capabilities across languages, highlighting performance gaps that cannot be bridged by simply scaling up model size. BenchMAX serves as a comprehensive multilingual evaluation platform, providing a promising test bed to promote the development of multilingual language models. The dataset and code are publicly accessible.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 37 of its 55 references that are in the map. First posted on arXiv 11 Feb 2025.

Builds on

37 of 55 references.

References · 37

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  10. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  11. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  12. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
25 more
  1. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  2. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  4. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  5. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  6. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  7. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  8. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  9. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  10. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  11. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  12. 2024InternLM2 Technical Reportcited by 391
  13. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  14. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  15. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 260
  16. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
  17. 2024Are We Done with MMLU?cited by 178
  18. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 156
  19. 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 121
  20. 2023Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Modelscited by 117
  21. 2023SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoningcited by 94
  22. 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 78
  23. 2024LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languagescited by 47
  24. 2024P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMscited by 29
  25. 2020Enhancing Answer Boundary Detection for Multilingual Machine Reading Comprehensioncited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 11.

23 papers.

Sort:

By area:

  1. 2025One ruler to measure them all: Benchmarking multilingual long-context language modelscited by 18
  2. 2025Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMscited by 17
  3. 2025Could Thinking Multilingually Empower LLM Reasoning?cited by 16
  4. 2025MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMscited by 16
  5. 2025MuBench: Assessment of Multilingual Capabilities of Large Language Models Across 61 Languagescited by 13
  6. 2025The Art of Asking: Multilingual Prompt Optimization for Synthetic Datacited by 5
  7. 2025Evaluating Large Language Model with Knowledge Oriented Language Specific Simple Question Answeringcited by 3
  8. 2025Adding LLMs to the psycholinguistic norming toolbox: A practical guide to getting the most out of human ratingscited by 3
  9. 2026Mending the Holes: Mitigating Reward Hacking in Reinforcement Learning for Multilingual Translationcited by 3
  10. 2026Cross-lingual Self-Consistency for Multilingual Reasoning with Language Modelscited by 3
  11. 2026Language as a Latent Variable for Reasoning Optimizationcited by 2
  12. 2026Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcyclingcited by 2
11 more
  1. 2025Uncovering inequalities in new knowledge learning by large language models across different languagescited by 1
  2. 2025LinguaMark: Do Multimodal Models Speak Fairly? A Benchmark-Based Evaluationcited by 1
  3. 2025LLaMAX2: Your Translation-Enhanced Model also Performs Well in Reasoningcited by 1
  4. 2025Designing Beyond Language: Sociotechnical Barriers in AI Health Technologies for Limited English Proficiencycited by 1
  5. 2026Language Models Entangle Language and Culturecited by 1
  6. 2026Lingua-SafetyBench: A Benchmark for Safety Evaluation of Multilingual Vision-Language Modelscited by 1
  7. 2026On-Policy Delta Distillation for Multilingual Math Reasoningcited by 1
  8. 2026English is Not All You Need: Systematically Exploring the Role of Multilinguality in LLM Post-Trainingcited by 0
  9. 2026Failure-Centered Runtime Evaluation for Deployed Trilingual Public-Space Agentscited by 0
  10. 2026The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modescited by 0
  11. 2026ADAGE: A Language-Agnostic Pipeline for Analogical Reasoning Evaluationcited by 0