mlsanityData to 13 Oct 2026

Process Supervision-Guided Policy Optimization for Code Generation

arXiv 2410.17621 ↗First posted 23 Oct 2024cs.AIcited by 25

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) with unit test feedback has enhanced large language models' (LLMs) code generation, but relies on sparse rewards provided only after complete code evaluation, limiting learning efficiency and incremental improvements. When generated code fails all unit tests, no learning signal is received, hindering progress on complex tasks. To address this, we propose a Process Reward Model (PRM) that delivers dense, line-level feedback on code correctness during generation, mimicking human code refinement and providing immediate guidance. We explore various strategies for training PRMs and integrating them into the RL framework, finding that using PRMs both as dense rewards and for value function initialization significantly boosts performance. Our experimental results also highlight the effectiveness of PRMs in enhancing RL-driven code generation, especially for long-horizon scenarios.

Key facts. Cited by 25 papers in the map (data through 13 Oct 2026). Builds on 25 of its 27 references that are in the map. First posted on arXiv 23 Oct 2024.

Builds on

25 of 27 references.

References · 25

  1. 2023GPT-4 Technical Reportcited by 18,278
  2. 2022Training language models to follow instructions with human feedbackcited by 15,856
  3. 2017Proximal Policy Optimization Algorithmscited by 14,592
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,205
  5. 2023Let's Verify Step by Stepcited by 3,672
  6. 2023Qwen Technical Reportcited by 3,310
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,197
  8. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,951
  9. 2022Competition-Level Code Generation with AlphaCodecited by 1,445
  10. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,173
  11. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 905
  12. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 743
13 more
  1. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 731
  2. 2023Textbooks Are All You Needcited by 566
  3. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 564
  4. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 432
  5. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 348
  6. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 298
  7. 2019SPoC: Search-based Pseudocode to Codecited by 176
  8. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 142
  9. 2023RLTF: Reinforcement Learning from Unit Test Feedbackcited by 112
  10. 2023Execution-based Code Generation using Deep Reinforcement Learningcited by 99
  11. 2024StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedbackcited by 75
  12. 2024Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervisioncited by 52
  13. 2024Learning Planning-based Reasoning by Trajectories Collection and Process Reward Synthesizingcited by 49

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 1; 2025: 9; 2026: 15.

25 papers.

Sort:

By area:

  1. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  2. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  3. 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluatorscited by 44
  4. 2025A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Modelscited by 38
  5. 2025Process-Supervised Reinforcement Learning for Code Generationcited by 22
  6. 2024Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generationcited by 21
  7. 2025Teaching Your Models to Understand Code via Focal Preference Alignmentcited by 10
  8. 2026PRPO: Aligning Process Reward with Outcome Reward in Policy Optimizationcited by 7
  9. 2026Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learningcited by 6
  10. 2025OpenGenAlign: A Preference Dataset and Benchmark for Trustworthy Reward Modeling in Open-Ended, Long-Context Generationcited by 5
  11. 2025Improving LLM-Generated Code Quality with GRPOcited by 5
  12. 2026Beyond Binary: Turning Partial Success into Dense Verifiable Rewards for Reinforcement Learning in Code Generationcited by 3
13 more
  1. 2025CRPE: Expanding The Reasoning Capability of Large Language Model for Code Generationcited by 2
  2. 2026RM -RF: Reward Model for Run-Free Unit Test Evaluationcited by 2
  3. 2026When Do Multi-Agent Systems Outperform? Analysing the Learning Efficiency of Agentic Systemscited by 2
  4. 2026From Failure to Feedback: Group Revision Unlocks Hard Cases in Object-Level Groundingcited by 2
  5. 2026DAJ: Data-Reweighted LLM Judge for Test-Time Scaling in Code Generationcited by 1
  6. 2026Rethinking Agentic Reinforcement Learning In Large Language Modelscited by 1
  7. 2026PRO-CUA: Process-Reward Optimization for Computer Use Agentscited by 1
  8. 2026X-Coder: Advancing Competitive Programming with Synthetic Tasks, Solutions, and Testscited by 0
  9. 2026Process Rewards with Learned Reliabilitycited by 0
  10. 2026EvoOptiGraph: Weakness-Driven Coevolution via Graph-Based Structural Generation for Optimization Modelingcited by 0
  11. 2026DHRCL:Training Code LLMs with Dense Hierarchical Rewards and Curriculum Learningcited by 0
  12. 2026Function-Level Execution Feedback for Code Preference Optimizationcited by 0
  13. 2026Performance Foundations of Parallel & Distributed Reasoning Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026FunPRM: Function-as-Step Process Reward Model with Meta Reward Correction for Code Generation
  2. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
  3. 2026Improving Small Language Models for Code Generation with Reinforcement Learning from Verification Feedback
  4. 2025Integrating Symbolic Execution into the Fine-Tuning of Code-Generating LLMs
  5. 2026Domain-Adaptable Reinforcement Learning for Code Generation with Dense Rewards
  6. 2026Beyond Execution: Static-Analysis Rewards and Hint-Conditioned Diffusion RL for Code Generation
  7. 2025CodeRL+: Improving Code Generation via Reinforcement with Execution Semantics Alignment
  8. 2026SWE-Shepherd: Advancing PRMs for Reinforcing Code Agents