# PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts

arXiv: https://arxiv.org/abs/2504.18428 · HTML: https://mlsanity.com/paper/2504.18428/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 33 of its 41 references that are in the map.
- First posted on arXiv 25 Apr 2025.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024)
- [Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs](https://mlsanity.com/paper/2412.21187/) (2024)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models](https://mlsanity.com/paper/2503.16419/) (2025)
- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://mlsanity.com/paper/2503.09567/) (2025)
- [Solving General Arithmetic Word Problems](https://mlsanity.com/paper/1608.01413/) (2016)
- [From System 1 to System 2: A Survey of Reasoning Large Language Models](https://mlsanity.com/paper/2502.17419/) (2025)
- [Chain of Draft: Thinking Faster by Writing Less](https://mlsanity.com/paper/2502.18600/) (2025)
- [Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models](https://mlsanity.com/paper/2410.07985/) (2024)
- [Token-Budget-Aware LLM Reasoning](https://mlsanity.com/paper/2412.18547/) (2024)
- [FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI](https://mlsanity.com/paper/2411.04872/) (2024)
- [mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval](https://mlsanity.com/paper/2407.19669/) (2024)
- [TheoremQA: A Theorem-driven Question Answering dataset](https://mlsanity.com/paper/2305.12524/) (2023)
- [Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMs](https://mlsanity.com/paper/2501.18585/) (2025)
- [Sampling-Efficient Test-Time Scaling: Self-Estimating the Best-of-N Sampling in Early Decoding](https://mlsanity.com/paper/2503.01422/) (2025)
- [Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observations](https://mlsanity.com/paper/2310.20246/) (2023)
- [Latent Space Chain-of-Embedding Enables Output-free LLM Self-Evaluation](https://mlsanity.com/paper/2410.13640/) (2024)
- [Linguistic Generalizability of Test-Time Scaling in Mathematical Reasoning](https://mlsanity.com/paper/2502.17407/) (2025)

## Built on by (most cited first, 30 of 56)

- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle](https://mlsanity.com/paper/2509.16679/) (2025)
- [Mitigating Overthinking in Large Reasoning Models via Manifold Steering](https://mlsanity.com/paper/2505.22411/) (2025)
- [A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models](https://mlsanity.com/paper/2509.03871/) (2025)
- [Think Natively: Unlocking Multilingual Reasoning with Consistency-Enhanced Reinforcement Learning](https://mlsanity.com/paper/2510.07300/) (2025)
- [MMATH: A Multilingual Benchmark for Mathematical Reasoning](https://mlsanity.com/paper/2505.19126/) (2025)
- [When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasoners](https://mlsanity.com/paper/2505.15257/) (2025)
- [Tiny Aya: Bridging Scale and Multilingual Depth](https://mlsanity.com/paper/2603.11510/) (2026)
- [Why Do Multilingual Reasoning Gaps Emerge in Reasoning Language Models?](https://mlsanity.com/paper/2510.27269/) (2025)
- [Understanding Multilingualism in Mixture-of-Experts LLMs: Routing Mechanism, Expert Specialization, and Layerwise Steering](https://mlsanity.com/paper/2601.14050/) (2026)
- [Crosslingual On-Policy Self-Distillation for Multilingual Reasoning](https://mlsanity.com/paper/2605.09548/) (2026)
- [mR3: Multilingual Rubric-Agnostic Reward Reasoning Models](https://mlsanity.com/paper/2510.01146/) (2025)
- [Unlocking Multilingual Reasoning Capability of LLMs and LVLMs through Representation Engineering](https://mlsanity.com/paper/2511.23231/) (2025)
- [DeepCompress: A Dual Reward Strategy for Dynamically Exploring and Compressing Reasoning Chains](https://mlsanity.com/paper/2510.27419/) (2025)
- [From KMMLU-Redux to KMMLU-Pro: A Professional Korean Benchmark Suite for LLM Evaluation](https://mlsanity.com/paper/2507.08924/) (2025)
- [GanitLLM: Difficulty-Aware Bengali Mathematical Reasoning through Curriculum-GRPO](https://mlsanity.com/paper/2601.06767/) (2026)
- [Multilingual Reasoning Gym: Multilingual Scaling of Procedural Reasoning Environments](https://mlsanity.com/paper/2603.10793/) (2026)
- [A Survey of Deep Learning for Geometry Problem Solving](https://mlsanity.com/paper/2507.11936/) (2025)
- [Language of Thought Shapes Output Diversity in Large Language Models](https://mlsanity.com/paper/2601.11227/) (2026)
- [Reasoning models do not yet follow their reasoning in autonomous driving: The KITScenes LongTail Dataset](https://mlsanity.com/paper/2603.23607/) (2026)
- [Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models](https://mlsanity.com/paper/2607.08186/) (2026)
- [MathReal: We Keep It Real! A Real Scene Benchmark for Evaluating Math Reasoning in Multimodal Large Language Models](https://mlsanity.com/paper/2508.06009/) (2025)
- [Gradual Code-Switching as Inference-Time Cross-Lingual Representational Alignment for LLMs](https://mlsanity.com/paper/2510.05678/) (2025)
- [The Reasoning Lingua Franca: A Double-Edged Sword for Multilingual AI](https://mlsanity.com/paper/2510.20647/) (2025)
- [MGSM-Pro: A Simple Strategy for Robust Multilingual Mathematical Reasoning Evaluation](https://mlsanity.com/paper/2601.21225/) (2026)
- [Unsupervised Confidence Calibration for Reasoning LLMs from a Single Generation](https://mlsanity.com/paper/2604.19444/) (2026)
- [LANG: Reinforcement Learning for Multilingual Reasoning with Language-Adaptive Hint Guidance](https://mlsanity.com/paper/2605.22567/) (2026)
- [Cross-lingual Self-Consistency for Multilingual Reasoning with Language Models](https://mlsanity.com/paper/2606.01464/) (2026)
- [UrduBench: An Urdu Reasoning Benchmark using Contextually Ensembled Translations with Human-in-the-Loop](https://mlsanity.com/paper/2601.21000/) (2026)

## Abstract (arXiv)

In this paper, we introduce PolyMath, a multilingual mathematical reasoning benchmark covering 18 languages and 4 easy-to-hard difficulty levels. Our benchmark ensures difficulty comprehensiveness, language diversity, and high-quality translation, making it a highly discriminative multilingual mathematical benchmark in the era of reasoning LLMs. We conduct a comprehensive evaluation for advanced LLMs and find that even Qwen-3-235B-A22B-Thinking and Gemini-2.5-pro, achieve only 54.6 and 52.2 benchmark scores, with about 40% accuracy under the highest level From a language perspective, our benchmark reveals several key challenges of LLMs in multilingual reasoning: (1) Reasoning performance varies widely across languages for current LLMs; (2) Input-output language consistency is low in reasoning LLMs and may be correlated with performance; (3) The thinking length differs significantly by language for current LLMs. Additionally, we demonstrate that controlling the output language in the instructions has the potential to affect reasoning performance, especially for some low-resource languages, suggesting a promising direction for improving multilingual capabilities in LLMs.