mlsanityData to 14 Oct 2026

PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts

arXiv 2504.18428 ↗First posted 25 Apr 2025cs.CLcited by 56

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

In this paper, we introduce PolyMath, a multilingual mathematical reasoning benchmark covering 18 languages and 4 easy-to-hard difficulty levels. Our benchmark ensures difficulty comprehensiveness, language diversity, and high-quality translation, making it a highly discriminative multilingual mathematical benchmark in the era of reasoning LLMs. We conduct a comprehensive evaluation for advanced LLMs and find that even Qwen-3-235B-A22B-Thinking and Gemini-2.5-pro, achieve only 54.6 and 52.2 benchmark scores, with about 40% accuracy under the highest level From a language perspective, our benchmark reveals several key challenges of LLMs in multilingual reasoning: (1) Reasoning performance varies widely across languages for current LLMs; (2) Input-output language consistency is low in reasoning LLMs and may be correlated with performance; (3) The thinking length differs significantly by language for current LLMs. Additionally, we demonstrate that controlling the output language in the instructions has the potential to affect reasoning performance, especially for some low-resource languages, suggesting a promising direction for improving multilingual capabilities in LLMs.

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 33 of its 41 references that are in the map. First posted on arXiv 25 Apr 2025.

Builds on

33 of 41 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI7 Nov
  2. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models10 Oct
  3. 2024Embedding Trajectory for Out-of-Distribution Detection in Mathematical Reasoning22 May
  4. 2023Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observations31 Oct
  5. 2022Language Models are Multilingual Chain-of-Thought Reasoners6 Oct

Other references · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2025Gemma 3 Technical Reportcited by 1,824
  10. 2025s1: Simple test-time scalingcited by 1,341
  11. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  12. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
16 more
  1. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 570
  2. 2025Humanity's Last Examcited by 491
  3. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 464
  4. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  5. 2016Solving General Arithmetic Word Problemscited by 360
  6. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  7. 2025Chain of Draft: Thinking Faster by Writing Lesscited by 297
  8. 2024Token-Budget-Aware LLM Reasoningcited by 247
  9. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 236
  10. 2023TheoremQA: A Theorem-driven Question Answering datasetcited by 192
  11. 2025Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMscited by 163
  12. 2025Sampling-Efficient Test-Time Scaling: Self-Estimating the Best-of-N Sampling in Early Decodingcited by 63
  13. 2024Latent Space Chain-of-Embedding Enables Output-free LLM Self-Evaluationcited by 57
  14. 2025Linguistic Generalizability of Test-Time Scaling in Mathematical Reasoningcited by 37
  15. 2025A Survey of Multilingual Reasoning in Language Modelscited by 33
  16. 2024P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMscited by 29

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 18; 2026: 38.

56 papers.

Sort:

By area:

  1. 2025Qwen3-VL Technical Reportcited by 2,793
  2. 2025Qwen3-Omni Technical Reportcited by 497
  3. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 35
  4. 2025Mitigating Overthinking in Large Reasoning Models via Manifold Steeringcited by 33
  5. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 25
  6. 2025Think Natively: Unlocking Multilingual Reasoning with Consistency-Enhanced Reinforcement Learningcited by 18
  7. 2025MMATH: A Multilingual Benchmark for Mathematical Reasoningcited by 17
  8. 2025When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasonerscited by 16
  9. 2026Tiny Aya: Bridging Scale and Multilingual Depthcited by 15
  10. 2025Why Do Multilingual Reasoning Gaps Emerge in Reasoning Language Models?cited by 12
  11. 2026Understanding Multilingualism in Mixture-of-Experts LLMs: Routing Mechanism, Expert Specialization, and Layerwise Steeringcited by 10
  12. 2026Crosslingual On-Policy Self-Distillation for Multilingual Reasoningcited by 10
38 more
  1. 2025mR3: Multilingual Rubric-Agnostic Reward Reasoning Modelscited by 8
  2. 2025Unlocking Multilingual Reasoning Capability of LLMs and LVLMs through Representation Engineeringcited by 8
  3. 2025DeepCompress: A Dual Reward Strategy for Dynamically Exploring and Compressing Reasoning Chainscited by 7
  4. 2025From KMMLU-Redux to KMMLU-Pro: A Professional Korean Benchmark Suite for LLM Evaluationcited by 6
  5. 2026GanitLLM: Difficulty-Aware Bengali Mathematical Reasoning through Curriculum-GRPOcited by 6
  6. 2026Multilingual Reasoning Gym: Multilingual Scaling of Procedural Reasoning Environmentscited by 5
  7. 2025A Survey of Deep Learning for Geometry Problem Solvingcited by 4
  8. 2026Language of Thought Shapes Output Diversity in Large Language Modelscited by 4
  9. 2026Reasoning models do not yet follow their reasoning in autonomous driving: The KITScenes LongTail Datasetcited by 4
  10. 2026Hidden Decoding at Scale: Latent Computation Scaling for Large Language Modelscited by 4
  11. 2025MathReal: We Keep It Real! A Real Scene Benchmark for Evaluating Math Reasoning in Multimodal Large Language Modelscited by 3
  12. 2025Gradual Code-Switching as Inference-Time Cross-Lingual Representational Alignment for LLMscited by 3
  13. 2025The Reasoning Lingua Franca: A Double-Edged Sword for Multilingual AIcited by 3
  14. 2026MGSM-Pro: A Simple Strategy for Robust Multilingual Mathematical Reasoning Evaluationcited by 3
  15. 2026Unsupervised Confidence Calibration for Reasoning LLMs from a Single Generationcited by 3
  16. 2026LANG: Reinforcement Learning for Multilingual Reasoning with Language-Adaptive Hint Guidancecited by 3
  17. 2026Cross-lingual Self-Consistency for Multilingual Reasoning with Language Modelscited by 3
  18. 2026UrduBench: An Urdu Reasoning Benchmark using Contextually Ensembled Translations with Human-in-the-Loopcited by 2
  19. 2026mAceReason-Math: A Dataset of High-Quality Multilingual Math Problems Ready For RLVRcited by 2
  20. 2026Round-Trip Translation Reveals What Frontier Multilingual Benchmarks Misscited by 2
  21. 2026Language as a Latent Variable for Reasoning Optimizationcited by 2
  22. 2026Efficient Multilingual Reasoning Transfer via Progressive Code-Switchingcited by 2
  23. 2025Language Confusion Gate: Language-Aware Decoding Through Model Self-Distillationcited by 1
  24. 2026Language Models Entangle Language and Culturecited by 1
  25. 2026On the Overscaling Curse of Parallel Thinking: System Efficacy Contradicts Sample Efficiencycited by 1
  26. 2026TriSpec: Ternary Speculative Decoding via Lightweight Proxy Verificationcited by 1
  27. 2026Valley3: Scaling Omni Foundation Models for E-commercecited by 1
  28. 2026On-Policy Delta Distillation for Multilingual Math Reasoningcited by 1
  29. 2026Towards Cross-lingual Values Judgment: A Consensus-Pluralism Perspectivecited by 0
  30. 2026Benchmarking EngGPT2-16B-A3B against Comparable Italian and International Open-source LLMscited by 0
  31. 2026Beyond Input Understanding: Diagnosing Multilingual Mathematical Reasoning with Directed Acyclic Trace Graphscited by 0
  32. 2026Multilingual Fine-Tuning via Localized Gradient Conflict Resolutioncited by 0
  33. 2026Distill Where the Student Goes: Teacher-Regularized RL for English-Evidence Cross-Lingual RAGcited by 0
  34. 2026PluraMath: Extending Mathematical Reasoning Evaluation Beyond High-Resource Languagescited by 0
  35. 2026Language Equality has a Price: A Systematic Investigation of Multi-turn LLM Performance for EU-24+cited by 0
  36. 2026K-EXAONE 2.0 Technical Reportcited by 0
  37. 2026RP-OPSD: Reasoning-Pivot-Guided On-Policy Self-Distillation for Multilingual Reasoning Transfercited by 0
  38. 2026Balance of Benchmarks: Semantic Density Reweighting for Task-Conditioned Model Comparisoncited by 0

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MathMist: A Parallel Multilingual Benchmark Dataset for Mathematical Problem Solving and Reasoning
  2. 2025MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMs
  3. 2024ConceptMath: A Bilingual Concept-wise Benchmark for Measuring Mathematical Reasoning of Large Language Models
  4. 2025Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gap
  5. 2025Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models
  6. 2025AI4Math: A Native Spanish Benchmark for University-Level Mathematical Reasoning in Large Language Models
  7. 2026Mathematical Reasoning in Large Language Models: Benchmarks, Architectures, Evaluation, and Open Challenges
  8. 2026LLM Parameters for Math Across Languages: Shared or Separate?