mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages?
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
We introduce mmPISA-bench, a compact high-quality multilingual reasoning benchmark derived from the OECD Programme for International Student Assessment (PISA). The benchmark consists of 25 multiple-choice questions that require reasoning in order to be answered correctly. Each question is provided in official human translations to 43 languages and complemented with machine-translated counterparts (i.e., 2,150 data points in total). We evaluate two mainstream proprietary LLMs across languages, reasoning effort levels, and translation types in terms of their ability to answer the questions correctly. Our results show that modern LLMs can reason effectively across all evaluated languages, achieve accuracy comparable to human test-takers, with some performance variations across covered languages. We further find that machine-translated questions do not degrade accuracy relative to official human translations which suggests that high-quality machine translation (synthetic data) might often be adequate for large-scale multilingual reasoning evaluations where official translations are not available. Finally, we analyze token usage and related inference cost and find that LLMs usage in some languages is simultaneously more expensive and less accurate.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 14 of its 18 references that are in the map. First posted on arXiv 5 Jun 2026.
Builds on
14 of 18 references.
Closest ancestors · 2
Same direction, newest first
- 2025When Models Reason in Your Language: Controlling Thinking Language Comes at the Cost of Accuracy28 May
- 2025When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasoners21 May
Other references · 12
- 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
- 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
- 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 214
- 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
- 2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 191
- 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 110
- 2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
- 2025Prompt Repetition Improves Non-Reasoning LLMscited by 27
- 2024Wait, that's not an option: LLMs Robustness with Incorrect Multiple-Choice Optionscited by 11
- 2025mSTEB: Massively Multilingual Evaluation of LLMs on Speech and Text Taskscited by 7
- 2025The AI Language Proficiency Monitor -- Tracking the Progress of LLMs on Multilingual Benchmarkscited by 7
- 2025PISA-Bench: The PISA Index as a Multilingual and Multimodal Metric for the Evaluation of Vision-Language Modelscited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMs
- 2025MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation
- 2024Eliciting Better Multilingual Structured Reasoning from LLMs through Code
- 2025Could Thinking Multilingually Empower LLM Reasoning?
- 2026Can LLM Reasoning Be Trusted? A Comparative Study: Using Human Benchmarking on Statistical Tasks
- 2025HellaSwag-Pro: A Large-Scale Bilingual Benchmark for Evaluating the Robustness of LLMs in Commonsense Reasoning
- 2024CLR-Bench: Evaluating Large Language Models in College-level Reasoning
- 2025PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts