mlsanityData to 14 Oct 2026

ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMs

arXiv 2506.18896 ↗First posted 23 Jun 2025cs.CLcited by 46

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Process Reward Models (PRMs) have recently emerged as a powerful framework for supervising intermediate reasoning steps in large language models (LLMs). Previous PRMs are primarily trained on model final output responses and struggle to evaluate intermediate thinking trajectories robustly, especially in the emerging setting of trajectory-response outputs generated by frontier reasoning models like Deepseek-R1. In this work, we introduce ReasonFlux-PRM, a novel trajectory-aware PRM explicitly designed to evaluate the trajectory-response type of reasoning traces. ReasonFlux-PRM incorporates both step-level and trajectory-level supervision, enabling fine-grained reward assignment aligned with structured chain-of-thought data. We adapt ReasonFlux-PRM to support reward supervision under both offline and online settings, including (i) selecting high-quality model distillation data for downstream supervised fine-tuning of smaller models, (ii) providing dense process-level rewards for policy optimization during reinforcement learning, and (iii) enabling reward-guided Best-of-N test-time scaling. Empirical results on challenging downstream benchmarks such as AIME, MATH500, and GPQA-Diamond demonstrate that ReasonFlux-PRM-7B selects higher quality data than strong PRMs (e.g., Qwen2.5-Math-PRM-72B) and human-curated baselines. Furthermore, our derived ReasonFlux-PRM-7B yields consistent performance improvements, achieving average gains of 12.1% in supervised fine-tuning, 4.5% in reinforcement learning, and 6.3% in test-time scaling. We also release our efficient ReasonFlux-PRM-1.5B for resource-constrained applications and edge deployment. Project: https://github.com/Gen-Verse/ReasonFlux

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 35 of its 69 references that are in the map. First posted on arXiv 23 Jun 2025.

Builds on

35 of 69 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  2. 2025Exploring the Limit of Outcome Reward for Learning Mathematical Reasoning10 Feb
  3. 2025Process Reinforcement through Implicit Rewards3 Feb
  4. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning13 Jan
  5. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization4 Jan
  6. 2024Process Reward Model with Q-Value Rankings15 Oct
  7. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning10 Oct
  8. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search6 Jun
  9. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
  10. 2022Solving math word problems with process- and outcome-based feedback25 Nov
  11. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 24

  1. 2024Qwen2 Technical Reportcited by 2,120
  2. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  3. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  4. 2024OpenAI o1 System Cardcited by 1,244
  5. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  6. 2025LIMO: Less is More for Reasoningcited by 498
  7. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 326
  8. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 313
  9. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 290
  10. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  11. 2024A Survey on Data Selection for Language Modelscited by 236
  12. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 217
12 more
  1. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 204
  2. 2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scalingcited by 136
  3. 2025Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoningcited by 119
  4. 2024Critique-out-Loud Reward Modelscited by 106
  5. 2024Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Modelscited by 99
  6. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 78
  7. 2025ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templatescited by 63
  8. 2025Co-Evolving LLM Coder and Unit Tester via Reinforcement Learningcited by 56
  9. 2024DOTS: Learning to Reason Dynamically in LLMs via Optimal Reasoning Trajectories Searchcited by 17
  10. 2024PromptIntern: Saving Inference Costs by Internalizing Recurrent Prompt during Large Language Model Fine-tuningcited by 13
  11. 2024Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Modelscited by 12
  12. 2024STEM-POM: Evaluating Language Models Math-Symbol Reasoning in Document Parsingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 18; 2026: 28.

46 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2026OpenClaw-RL: Train Any Agent Simply by Talkingcited by 90
  3. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  4. 2025Agentic Reinforcement Learning with Implicit Step Rewardscited by 47
  5. 2025Demystifying Reinforcement Learning in Agentic Reasoningcited by 33
  6. 2025Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Trainingcited by 29
  7. 2025Hard2Verify: A Step-Level Verification Benchmark for Open-Ended Frontier Mathcited by 22
  8. 2026SOD: Step-wise On-policy Distillation for Small Language Model Agentscited by 21
  9. 2025Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Modelscited by 13
  10. 2026Which Reasoning Trajectories Teach Students to Reason Better? A Simple Metric of Informative Alignmentcited by 13
  11. 2025Good Learners Think Their Thinking: Generative PRM Makes Large Reasoning Model More Efficient Math Learnercited by 11
  12. 2026Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoningcited by 11
34 more
  1. 2026Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaborationcited by 11
  2. 2025Hybrid Reward Normalization for Process-supervised Non-verifiable Agentic Taskscited by 10
  3. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 9
  4. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  5. 2025Fin-PRM: A Domain-Specialized Process Reward Model for Financial Reasoning in Large Language Modelscited by 7
  6. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 7
  7. 2025StructVRM: Aligning Multimodal Reasoning with Structured and Verifiable Reward Modelscited by 6
  8. 2025NIRVANA: Structured Pruning Reimagined for Large Language Model Compressioncited by 6
  9. 2026Retrospective Progress-Aware Self-Refinement for LLM Agent Trainingcited by 6
  10. 2026From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AIcited by 6
  11. 2025CoLD: Counterfactually-Guided Length Debiasing for Process Reward Models in Mathematical Reasoningcited by 5
  12. 2025TaTToo: Tool-Grounded Thinking PRM for Test-Time Scaling in Tabular Reasoningcited by 5
  13. 2025PRInTS: Reward Modeling for Long-Horizon Information Seekingcited by 5
  14. 2026Characterizing, Evaluating, and Optimizing Complex Reasoningcited by 5
  15. 2026Rewarding the Scientific Process: Process-Level Reward Modeling for Agentic Data Analysiscited by 5
  16. 2026ReasoningFlow: Discourse Structures for Understanding LLM Reasoning Tracescited by 5
  17. 2026From Absolute to Relative: Rethinking Reward Shaping in Group-Based Reinforcement Learningcited by 4
  18. 2026What If We Allocate Test-Time Compute Adaptively?cited by 4
  19. 2025TrajSelector: Harnessing Latent Representations for Efficient and Effective Best-of-N in Large Reasoning Modelcited by 3
  20. 2026GraphPO: Graph-based Policy Optimization for Reasoning Modelscited by 3
  21. 2025Curing Miracle Steps in LLM Mathematical Reasoning with Rubric Rewardscited by 1
  22. 2026Incentivizing Cardiologist-Like Reasoning in MLLMs for Interpretable Echocardiographic Diagnosiscited by 1
  23. 2026VeriBound: PAC-Bayesian Generalization Bounds for Process Reward Models Trained with Formal Verification Toolscited by 1
  24. 2025Stepwise Think-Critique: Interleaved Reasoning and Self-Critique in a Single LLMcited by 0
  25. 2026SCATR: Simple Calibrated Test-Time Rankingcited by 0
  26. 2026Characterizing Model-Native Skillscited by 0
  27. 2026Structural Rationale Distillation via Reasoning Space Compressioncited by 0
  28. 2026LC-ERD: Mining Latent Logic for Self-Evolving Reasoning via Consistency-Regulated Reward Decompositioncited by 0
  29. 2026Not only where, But when: Temporal Scheduling for RLVRcited by 0
  30. 2026Rewarding Better Thinking for LLM Preference Alignmentcited by 0
  31. 2026Reasoning Jury: Multi-Model Consensus for Evaluating Reasoning Tracescited by 0
  32. 2026DCRL: Decoupling and Coupling Reinforcement Learning via Policy-Reward Manifold Alignmentcited by 0
  33. 2026Advancing Entropy-Level Credit Assignment in RLVR via Proximal Entropy Policy Optimizationcited by 0
  34. 2026Unlocking Token Rewards via Training-Free Reward Attributioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Models
  2. 2026LLM Reasoning with Process Rewards for Outcome-Guided Steps
  3. 2025Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning
  4. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoning
  5. 2026Process Reward Models Meet Planning: Generating Precise and Scalable Datasets for Step-Level Rewards
  6. 2025R-PRM: Reasoning-Driven Process Reward Modeling
  7. 2025PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models
  8. 2025Rectifying LLM Thought from Lens of Optimization