mlsanityData to 14 Oct 2026

Linguistic Generalizability of Test-Time Scaling in Mathematical Reasoning

arXiv 2502.17407 ↗First posted 24 Feb 2025cs.CLcited by 37

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Scaling pre-training compute has proven effective for achieving mulitlinguality, but does the same hold for test-time scaling? In this work, we introduce MCLM, a multilingual math benchmark featuring competition-level problems in 55 languages. We test three test-time scaling methods-Outcome Reward Modeling (ORM), Process Reward Modeling (ORM), and Budget Forcing (BF)-on both Qwen2.5-1.5B Math and MR1-1.5B, a multilingual LLM we trained for extended reasoning. Our experiments show that using Qwen2.5-1.5B Math with ORM achieves a score of 35.8 on MCLM, while BF on MR1-1.5B attains 35.2. Although "thinking LLMs" have recently garnered significant attention, we find that their performance is comparable to traditional scaling methods like best-of-N once constrained to similar levels of inference FLOPs. Moreover, while BF yields a 20-point improvement on English AIME, it provides only a 1.94-point average gain across other languages-a pattern consistent across the other test-time scaling methods we studied-higlighting that test-time scaling may not generalize as effectively to multilingual tasks. To foster further research, we release MCLM, MR1-1.5B, and evaluation results.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 51 of its 80 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

51 of 80 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Competitive Programming with Large Reasoning Models3 Feb
  2. 2025s1: Simple test-time scaling31 Jan
  3. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs30 Dec
  4. 2024O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?25 Nov
  5. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing18 Apr
  6. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training29 Sep

Other references · 45

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2024GPT-4o System Cardcited by 4,588
  10. 2023Let's Verify Step by Stepcited by 3,789
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  12. 2024OpenAI o1 System Cardcited by 1,244
33 more
  1. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  2. 2023PaLM 2 Technical Reportcited by 1,162
  3. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  4. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  5. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  6. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  7. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  8. 2025Process Reinforcement through Implicit Rewardscited by 504
  9. 2025LIMO: Less is More for Reasoningcited by 498
  10. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 441
  11. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 415
  12. 2023Llemma: An Open Language Model For Mathematicscited by 388
  13. 20242 OLMo 2 Furiouscited by 340
  14. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 313
  15. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 284
  16. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 190
  17. 2025When More is Less: Understanding Chain-of-Thought Length in LLMscited by 183
  18. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 147
  19. 2024Liger Kernel: Efficient Triton Kernels for LLM Trainingcited by 130
  20. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 123
  21. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 109
  22. 2024KMMLU: Measuring Massive Multitask Language Understanding in Koreancited by 91
  23. 2024AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modelingcited by 77
  24. 2021Scaling Scaling Laws with Board Gamescited by 63
  25. 2023Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observationscited by 59
  26. 2025Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gapcited by 43
  27. 2024Question Translation Training for Better Multilingual Reasoningcited by 42
  28. 2023PLUG: Leveraging Pivot Language in Cross-Lingual Instruction Tuningcited by 39
  29. 2024mCoT: Multilingual Instruction Tuning for Reasoning Consistency in Language Modelscited by 33
  30. 2025On the Emergence of Thinking in LLMs I: Searching for the Right Intuitioncited by 22
  31. 2024Language Models as Compilers: Simulating Pseudocode Execution Improves Algorithmic Reasoning in Language Modelscited by 20
  32. 2024Spanish and LLM Benchmarks: is MMLU Lost in Translation?cited by 20
  33. 2024Cross-lingual Transfer of Reward Models in Multilingual Alignmentcited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 18; 2026: 19.

37 papers.

Sort:

By area:

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2025PolyMath: Evaluating Mathematical Reasoning in Multilingual Contextscited by 56
  3. 2025Crosslingual Reasoning through Test-Time Scalingcited by 46
  4. 2026When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturationcited by 33
  5. 2025When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Researchcited by 26
  6. 2025MMATH: A Multilingual Benchmark for Mathematical Reasoningcited by 17
  7. 2025When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasonerscited by 16
  8. 2025MultiFinBen: Benchmarking Large Language Models for Multilingual and Multimodal Financial Applicationcited by 14
  9. 2025Scaling Evaluation-time Compute with Reasoning Models as Evaluatorscited by 13
  10. 2025Why Do Multilingual Reasoning Gaps Emerge in Reasoning Language Models?cited by 12
  11. 2026Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillationcited by 11
  12. 2025Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thoughtcited by 10
25 more
  1. 2026Understanding Multilingualism in Mixture-of-Experts LLMs: Routing Mechanism, Expert Specialization, and Layerwise Steeringcited by 10
  2. 2025When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMscited by 9
  3. 2025Beyond English-Centric Training: How Reinforcement Learning Improves Cross-Lingual Reasoning in LLMscited by 8
  4. 2025mR3: Multilingual Rubric-Agnostic Reward Reasoning Modelscited by 8
  5. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 7
  6. 2026GanitLLM: Difficulty-Aware Bengali Mathematical Reasoning through Curriculum-GRPOcited by 6
  7. 2026Gained in Translation: Privileged Pairwise Judges Enhance Multilingual Reasoningcited by 6
  8. 2025Test-Time Scaling of Reasoning Models for Machine Translationcited by 5
  9. 2026Multilingual Reasoning Gym: Multilingual Scaling of Procedural Reasoning Environmentscited by 5
  10. 2026Language of Thought Shapes Output Diversity in Large Language Modelscited by 4
  11. 2025Controlling Language Confusion in Multilingual LLMscited by 3
  12. 2025Less Data Less Tokens: Multilingual Unification Learning for Efficient Test-Time Reasoning in LLMscited by 3
  13. 2025Test-Time Policy Adaptation for Enhanced Multi-Turn Interactions with LLMscited by 3
  14. 2026Round-Trip Translation Reveals What Frontier Multilingual Benchmarks Misscited by 2
  15. 2026x1: Learning to Think Adaptively Across Languages and Culturescited by 2
  16. 2026What Should a Self-Teacher See? Privileged Context Design for On-Policy Self-Distillationcited by 2
  17. 2026Judging What We Cannot Solve: A Consequence-Based Approach for Oracle-Free Evaluation of Research-Level Mathcited by 1
  18. 2026Words & Weights: Streamlining Multi-Turn Interactions via Co-Adaptationcited by 1
  19. 2025Won: Establishing Best Practices for Korean Financial NLPcited by 0
  20. 2026When to Vote, When to Rewrite: Disagreement-Guided Strategy Routing for Test-Time Scalingcited by 0
  21. 2026Does CoT-Pass@k Really Check the CoT? A Multilingual Mathematical Auditcited by 0
  22. 2026Language-Conditioned Token and Reasoning Efficiency in Large Language Models: A Paired Cross-Lingual Study Protocolcited by 0
  23. 2026Cross-lingual Calibration of Pre-Generation Success Probes for Multilingual LLM Routingcited by 0
  24. 2026North or South? Diagnosing LLMs' Understanding of North Korean Language, Society, and Knowledgecited by 0
  25. 2026Scaling Multilingual Safety and Reasoning Evaluation through Reasoning-Guided Translationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking Reward Models for Multi-Domain Test-Time Scaling
  2. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?
  3. 2025Best-of-L: Cross-Lingual Reward Modeling for Mathematical Reasoning
  4. 2025Multilingual Test-Time Scaling via Initial Thought Transfer
  5. 2025Inference-Time Scaling for Complex Tasks: Where We Stand and What Lies Ahead
  6. 2024Benchmarking Large Language Models for Math Reasoning Tasks
  7. 2025MathMist: A Parallel Multilingual Benchmark Dataset for Mathematical Problem Solving and Reasoning
  8. 2025Think Twice: Enhancing LLM Reasoning by Scaling Multi-round Test-time Thinking