mlsanityData to 14 Oct 2026

Reasoning over mathematical objects: on-policy reward modeling and test time aggregation

arXiv 2603.18886 ↗First posted 19 Mar 2026cs.AI cs.CLcited by 3

On the map: LLM reasoning → Mathematical reasoning

also in 1 more
Abstract (arXiv)

The ability to precisely derive mathematical objects is a core requirement for downstream STEM applications, including mathematics, physics, and chemistry, where reasoning must culminate in formally structured expressions. Yet, current LM evaluations of mathematical and scientific reasoning rely heavily on simplified answer formats such as numerical values or multiple choice options due to the convenience of automated assessment. In this paper we provide three contributions for improving reasoning over mathematical objects: (i) we build and release training data and benchmarks for deriving mathematical objects, the Principia suite; (ii) we provide training recipes with strong LLM-judges and verifiers, where we show that on-policy judge training boosts performance; (iii) we show how on-policy training can also be used to scale test-time compute via aggregation. We find that strong LMs such as Qwen3-235B and o3 struggle on Principia, while our training recipes can bring significant improvements over different LLM backbones, while simultaneously improving results on existing numerical and MCQA tasks, demonstrating cross-format generalization of reasoning abilities.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 81 of its 142 references that are in the map. First posted on arXiv 19 Mar 2026.

Builds on

81 of 142 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning27 Nov
  2. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitions29 May
  3. 2025RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics18 May
  4. 2025AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset23 Apr
  5. 2025NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questions18 Feb
  6. 2024AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modeling19 Dec
  7. 2024Orca-Math: Unlocking the potential of SLMs in Grade School Math16 Feb
  8. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models5 Feb
  9. 2023ARB: Advanced Reasoning Benchmark for Large Language Models25 Jul

Other references · 72

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2024GPT-4o System Cardcited by 4,588
  9. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  12. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
38 more
  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  2. 2016Concrete Problems in AI Safetycited by 1,642
  3. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  4. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  5. 2024OpenAI o1 System Cardcited by 1,244
  6. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,071
  7. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  8. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  9. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  10. 2024Self-Rewarding Language Modelscited by 681
  11. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  12. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  13. 2025Kimi K2: Open Agentic Intelligencecited by 567
  14. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 511
  15. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  16. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 461
  17. 2025The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Searchcited by 422
  18. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  19. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 377
  20. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 314
  21. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
  22. 2023Universal Self-Consistency for Large Language Model Generationcited by 203
  23. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 200
  24. 2025Skywork Open Reasoner 1 Technical Reportcited by 165
  25. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 150
  26. 2019On the Measure of Intelligencecited by 144
  27. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 138
  28. 2025General-Reasoner: Advancing LLM Reasoning Across All Domainscited by 135
  29. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scalingcited by 113
  30. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 87
  31. 2025AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learningcited by 87
  32. 2025SPICE: Self-Play In Corpus Environments Improves Reasoningcited by 83
  33. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 80
  34. 2025SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplinescited by 79
  35. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 75
  36. 2025Tongyi DeepResearch Technical Reportcited by 66
  37. 2025Parallel-R1: Towards Parallel Thinking via Reinforcement Learningcited by 61
  38. 2025MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoningcited by 59

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Autodata: An agentic data scientist to create high quality synthetic datacited by 16
  2. 2026Shorter Reasoning, Earlier Answers? An Evaluation of Reasoning Interfacescited by 1
  3. 2026MathForm: Scaling Mathematical Autoformalization with Knowledge Retrieval and Verification-Guided Refinementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking Math Reasoning Evaluation: A Robust LLM-as-a-Judge Framework Beyond Symbolic Rigidity
  2. 2025VAR-MATH: Probing True Mathematical Reasoning in LLMS via Symbolic Multi-Instance Benchmarks
  3. 2026LiveMathematicianBench: A Live Benchmark for Research-Level Mathematical Reasoning with Proof Sketches
  4. 2026Mathematical Reasoning in Large Language Models: Benchmarks, Architectures, Evaluation, and Open Challenges
  5. 2025Thinking Machines: Mathematical Reasoning in the Age of LLMs
  6. 2026FABSVer: Faster Training and Better Self-Verification for LLM Mathematical Reasoning
  7. 2025Step-KTO: Optimizing Mathematical Reasoning through Stepwise Binary Feedback
  8. 2025Atomic Thinking of LLMs: Decoupling and Exploring Mathematical Reasoning Abilities