mlsanityData to 7 Oct 2026

P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs

arXiv 2411.09116 ↗First posted 14 Nov 2024cs.CLcited by 29

Not on the map. Why?

Abstract (arXiv)

Recent advancements in large language models (LLMs) showcase varied multilingual capabilities across tasks like translation, code generation, and reasoning. Previous assessments often limited their scope to fundamental natural language processing (NLP) or isolated capability-specific tasks. To alleviate this drawback, we aim to present a comprehensive multilingual multitask benchmark. First, we introduce P-MMEval, a large-scale benchmark covering effective fundamental and capability-specialized datasets. Furthermore, P-MMEval delivers consistent language coverage across various datasets and provides parallel samples. Finally, we conduct extensive experiments on representative multilingual model series to compare performances across models and tasks, explore the relationship between multilingual performances and factors such as tasks, model sizes, languages, and prompts, and examine the effectiveness of knowledge transfer from English to other languages. The resulting insights are intended to offer valuable guidance for future research. The dataset is available at https://huggingface.co/datasets/Qwen/P-MMEval.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 34 of its 46 references that are in the map. First posted on arXiv 14 Nov 2024.

Builds on

34 of 46 references.

References · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2023Qwen Technical Reportcited by 3,273
  12. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
22 more
  1. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  2. 2024Qwen2 Technical Reportcited by 2,076
  3. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  4. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  5. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  6. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  7. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  8. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  9. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  10. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  11. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  12. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  13. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  14. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  15. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 247
  16. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 221
  17. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
  18. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 167
  19. 2024HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalizationcited by 84
  20. 2021It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoningcited by 69
  21. 2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
  22. 2020XL-WiC: A Multilingual Benchmark for Evaluating Semantic Contextualizationcited by 42

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 20; 2026: 8.

29 papers.

Sort:

By area:

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2024Qwen2.5 Technical Reportcited by 4,713
  3. 2025Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Modelscited by 65
  4. 2025PolyMath: Evaluating Mathematical Reasoning in Multilingual Contextscited by 52
  5. 2025A Survey of Multilingual Reasoning in Language Modelscited by 31
  6. 2025BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Modelscited by 23
  7. 2025BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Datasetcited by 11
  8. 2025Why Do Multilingual Reasoning Gaps Emerge in Reasoning Language Models?cited by 11
  9. 2026Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillationcited by 11
  10. 2025Mind the Gap... or Not? How Translation Errors and Evaluation Details Skew Multilingual Resultscited by 10
  11. 2025From Neurons to Semantics: Evaluating Cross-Linguistic Alignment Capabilities of Large Language Models via Neurons Alignmentcited by 9
  12. 2025From KMMLU-Redux to KMMLU-Pro: A Professional Korean Benchmark Suite for LLM Evaluationcited by 6
17 more
  1. 2026Translation as a Scalable Proxy for Multilingual Evaluationcited by 6
  2. 2026OpenCompass: A Universal Evaluation Platform for Large Language Modelscited by 5
  3. 2025MiLiC-Eval: Benchmarking Multilingual LLMs for China's Minority Languagescited by 4
  4. 2025Benchmarking and Boosting Multilingual Capabilities of LVLMs via OCR-Centric Reinforcement Learningcited by 4
  5. 2025Enhancing LLM Language Adaption through Cross-lingual In-Context Pre-trainingcited by 4
  6. 2025Aligning Multilingual Reasoning with Verifiable Semantics from a High-Resource Expert Modelcited by 4
  7. 2025PISA-Bench: The PISA Index as a Multilingual and Multimodal Metric for the Evaluation of Vision-Language Modelscited by 3
  8. 2025MULTITAT: Benchmarking Multilingual Table-and-Text Question Answeringcited by 2
  9. 2025SASFT: Sparse Autoencoder-guided Supervised Finetuning to Mitigate Unexpected Code-Switching in LLMscited by 2
  10. 2025Evaluating Modern Large Language Models on Low-Resource and Morphologically Rich Languages:A Cross-Lingual Benchmark Across Cantonese, Japanese, and Turkishcited by 2
  11. 2025Uncovering inequalities in new knowledge learning by large language models across different languagescited by 1
  12. 2025MEDAL: A Framework for Benchmarking LLMs as Multilingual Open-Domain Dialogue Evaluatorscited by 1
  13. 2026Extending Item Response Theory for Efficient and Meaningful Multilingual Evaluationcited by 1
  14. 2026Towards Cross-lingual Values Judgment: A Consensus-Pluralism Perspectivecited by 0
  15. 2026The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modescited by 0
  16. 2026Who Pays More for Safety? Measuring the Disparate Cost of Safety Alignment across Languagescited by 0
  17. 2026VFA: Empowering Multilingual MLLMs via Vision-Free Adaptationcited by 0