mlsanityData to 14 Oct 2026

Process-based Self-Rewarding Language Models

arXiv 2503.03746 ↗First posted 5 Mar 2025cs.CL cs.AIcited by 22

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Large Language Models have demonstrated outstanding performance across various downstream tasks and have been widely applied in multiple scenarios. Human-annotated preference data is used for training to further improve LLMs' performance, which is constrained by the upper limit of human performance. Therefore, Self-Rewarding method has been proposed, where LLMs generate training data by rewarding their own outputs. However, the existing self-rewarding paradigm is not effective in mathematical reasoning scenarios and may even lead to a decline in performance. In this work, we propose the Process-based Self-Rewarding pipeline for language models, which introduces long-thought reasoning, step-wise LLM-as-a-Judge, and step-wise preference optimization within the self-rewarding paradigm. Our new paradigm successfully enhances the performance of LLMs on multiple mathematical reasoning benchmarks through iterative Process-based Self-Rewarding, demonstrating the immense potential of self-rewarding to achieve LLM reasoning that may surpass human capabilities.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 27 of its 37 references that are in the map. First posted on arXiv 5 Mar 2025.

Builds on

27 of 37 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  2. 2023Let's Verify Step by Step31 May
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  4. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2024A Survey on LLM-as-a-Judgecited by 1,423
  8. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  9. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  10. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  11. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  12. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
11 more
  1. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  2. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  3. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 388
  4. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 273
  5. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
  6. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 149
  7. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 134
  8. 2023Answering Questions by Meta-Reasoning over Multiple Chains of Thoughtcited by 87
  9. 2024MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipelinecited by 81
  10. 2024Step-level Value Preference Optimization for Mathematical Reasoningcited by 72
  11. 2024EDT: Improving Large Language Models' Generation by Entropy-based Dynamic Temperature Samplingcited by 45

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 17; 2026: 5.

22 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  2. 2025Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoningcited by 46
  3. 2025SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoningcited by 42
  4. 2025R-PRM: Reasoning-Driven Process Reward Modelingcited by 38
  5. 2025A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Modelscited by 38
  6. 2025Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigationcited by 37
  7. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 15
  8. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  9. 2025SSR-Zero: Simple Self-Rewarding Reinforcement Learning for Machine Translationcited by 13
  10. 2025RoT: Enhancing Table Reasoning with Iterative Row-Wise Traversalscited by 10
  11. 2025How Does Alignment Enhance LLMs' Multilingual Capabilities? A Language Neurons Perspectivecited by 8
  12. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
10 more
  1. 2025PATS: Process-Level Adaptive Thinking Mode Switchingcited by 6
  2. 2025Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Futurecited by 5
  3. 2025SSPO: Self-traced Step-wise Preference Optimization for Process Supervision and Reasoning Compressioncited by 2
  4. 2026Towards Robust Process Reward Modeling via Noise-aware Learningcited by 2
  5. 2025Enhancing Speech Large Language Models through Reinforced Behavior Alignmentcited by 1
  6. 2026Confidence-Aware Alignment Makes Reasoning LLMs More Reliablecited by 1
  7. 2026J-Zero: Unified Challenger--Solver--Judge Self-Evolution from Zero Datacited by 1
  8. 2025SDGO: Self-Discrimination-Guided Optimization for Consistent Safety in Large Language Modelscited by 0
  9. 2026Reflective Recovery: A Self-Supervised Method for Reasoning by Learning from Mistakescited by 0
  10. 2026Beyond Binary Verifiable Rewards: Behavioral Self-Rewarding for SWE Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Self-rewarding correction for mathematical reasoning
  2. 2025Accelerating LLM Reasoning via Early Rejection with Partial Reward Modeling
  3. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoning
  4. 2024Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewards
  5. 2024Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoning
  6. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning
  7. 2025A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning
  8. 2023Let's Reinforce Step by Step