Data to 14 Oct 2026

Step-level Value Preference Optimization for Mathematical Reasoning

arXiv 2406.10858 ↗First posted 16 Jun 2024cs.CL cs.AIcited by 72

On the map: LLM reasoning → Mathematical reasoning

also in 1 more

LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

Direct Preference Optimization (DPO) using an implicit reward model has proven to be an effective alternative to reinforcement learning from human feedback (RLHF) for fine-tuning preference aligned large language models (LLMs). However, the overall preference annotations of responses do not fully capture the fine-grained quality of model outputs in complex multi-step reasoning tasks, such as mathematical reasoning. To address this limitation, we introduce a novel algorithm called Step-level Value Preference Optimization (SVPO). Our approach employs Monte Carlo Tree Search (MCTS) to automatically annotate step-level preferences for multi-step reasoning. Furthermore, from the perspective of learning-to-rank, we train an explicit value model to replicate the behavior of the implicit reward model, complementing standard preference optimization. This value model enables the LLM to generate higher reward responses with minimal cost during inference. Experimental results demonstrate that our method achieves state-of-the-art performance on both in-domain and out-of-domain mathematical reasoning benchmarks. Our code is available at urlhttps://github.com/MARIO-Math-Reasoning/SuperMARIO.

Key facts. Cited by 72 papers in the map (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 27 of its 42 references that are in the map. First posted on arXiv 16 Jun 2024.

Builds on

27 of 42 references.

Closest ancestors · 8

Same direction, newest first

Other references · 19

2022Training language models to follow instructions with human feedbackcited by 15,971
2017Proximal Policy Optimization Algorithmscited by 15,215
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
2021Training Verifiers to Solve Math Word Problemscited by 10,109
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
2023Qwen Technical Reportcited by 3,344
2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562

7 more

2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
2024AlphaMath Almost Zero: Process Supervision without Processcited by 162
2024Noise Contrastive Alignment of Language Models with Explicit Rewardscited by 85
2024MARIO Eval: Evaluate Your Math LLM with your Math LLM--A mathematical dataset evaluation toolkitcited by 14
2023MPrompt: Exploring Multi-level Prompt Tuning for Machine Reading Comprehensioncited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 13; 2025: 45; 2026: 14.

72 papers.

Sort:

By area:

2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 301
2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 293
2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
2026Self-Distilled RLVRcited by 131
2025What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secretcited by 98
2024LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoningcited by 91
2024Process Reward Model with Q-Value Rankingscited by 80
2025STAIR: Improving Safety Alignment with Introspective Reasoningcited by 77
2025Self-rewarding correction for mathematical reasoningcited by 70
2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
2025RealSafe-R1: Safety-Aligned DeepSeek-R1 without Compromising Reasoning Capabilitycited by 59
2024Offline Reinforcement Learning for LLM Multi-Step Reasoningcited by 54

38 more

2024Entropy-Regularized Process Reward Modelcited by 38
2025A Survey of Direct Preference Optimizationcited by 36
2025Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoningcited by 34
2024Towards Self-Improvement of LLMs via MCTS: Leveraging Stepwise Knowledge with Curriculum Preference Learningcited by 30
2025Qwen Look Again: Guiding Vision-Language Reasoning Models to Re-attention Visual Informationcited by 28
2025Exploiting Tree Structure for Credit Assignment in RL Training of LLMscited by 27
2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
2024Selective Preference Optimization via Token-Level Reward Function Estimationcited by 25
2025Thinking Machines: A Survey of LLM based Reasoning Strategiescited by 25
2025Optimas: Optimizing Compound AI Systems with Globally Aligned Local Rewardscited by 24
2024Enhancing Multi-Step Reasoning Abilities of Language Models through Direct Q-Function Optimizationcited by 22
2025Process-based Self-Rewarding Language Modelscited by 22
2025BoostStep: Boosting mathematical capability of Large Language Models via improved single-step reasoningcited by 21
2025KnowRL: Exploring Knowledgeable Reinforcement Learning for Factualitycited by 20
2025ReForm: Reflective Autoformalization with Prospective Bounded Sequence Optimizationcited by 20
2024Markov Chain of Thought for Efficient Mathematical Reasoningcited by 19
2025Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuningcited by 19
2024Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage Policy Optimizationcited by 17
2024Learning Evolving Tools for Large Language Modelscited by 16
2025DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimizationcited by 16
2025CheMatAgent: Enhancing LLMs for Chemistry and Materials Science through Tree-Search Based Tool Learningcited by 15
2025Full-Step-DPO: Self-Supervised Preference Optimization with Step-wise Rewards for Mathematical Reasoningcited by 14
2024BEATS: Optimizing LLM Mathematical Capabilities with BackVerify and Adaptive Disambiguate based Efficient Tree Searchcited by 13
2025Route-and-Reason: Scaling Large Language Model Reasoning with Reinforced Model Routercited by 13
2026What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agentscited by 13
2025An Efficient and Precise Training Data Construction Framework for Process-supervised Reward Model in Mathematical Reasoningcited by 12
2025Self-Guided Process Reward Optimization with Redefined Step-wise Advantage for Process Reinforcement Learningcited by 11
2025Good Learners Think Their Thinking: Generative PRM Makes Large Reasoning Model More Efficient Math Learnercited by 11
2025ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimizationcited by 10
2025GroundedPRM: Tree-Guided and Fidelity-Aware Process Reward Modeling for Step-Level Reasoningcited by 10
2026Reinforcement Learning from Rich Feedback with Distributional DAggercited by 10
2025CMCTS: A Constrained Monte Carlo Tree Search Framework for Mathematical Reasoning in Large Language Modelcited by 9
2025Reveal the Mystery of DPO: The Connection between DPO and RL Algorithmscited by 8
2025Iterative Deepening Sampling as Efficient Test-Time Scalingcited by 8
2025Two Heads Are Better Than One: Dual-Model Verbal Reflection at Inference-Timecited by 8
2026Self-Distilled Policy Gradientcited by 8
2025Direct Value Optimization: Improving Chain-of-Thought Reasoning in LLMs with Refined Valuescited by 6
2025Efficient Pretraining Length Scalingcited by 6

50 most cited of 72 All 72 (JSON)

Similar, not cited

Close in wording, no citation either way