mlsanityData to 14 Oct 2026

MURPHY: Feedback-Aware GRPO with Retrospective Credit Assignment for Multi-Turn Code Generation

arXiv 2511.07833 ↗First posted 11 Nov 2025cs.LG cs.AIcited by 5

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) has become a standard recipe for post-training LLMs on reasoning tasks, with Group Relative Policy Optimization (GRPO) emerging as a leading approach. However, GRPO and its variants are inherently single-turn: they optimize from terminal rewards on isolated prompt-response pairs, leaving them poorly suited to agentic settings where models must iteratively refine solutions in response to environmental feedback. We introduce MURPHY, a multi-turn extension of GRPO for self-correcting code generation. MURPHY constructs feedback-conditioned rollout trees in which failed candidate solutions are paired with executor feedback and expanded into subsequent turns, and propagates rewards backward through the tree so that later successful refinements credit earlier attempts that surfaced informative feedback. We study two propagation strategies, Max Reward (MARS) and Mean Reward (MERS), and introduce post-rollout pruning mechanisms that reduce multi-turn optimization cost. Across three code generation benchmarks (HumanEval, MBPP, LiveCodeBench-v6) and two model families (Qwen3-1.7B/4B, OLMo-2-7B), MURPHY delivers up to 6% absolute pass@1 gains over the strongest prior multi-turn execution-feedback methods. Gains are largest on the Medium/Hard subset (+4.38/+4.20 at Iter-5), where iterative self-correction matters more.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 16 of its 39 references that are in the map. First posted on arXiv 11 Nov 2025.

Builds on

16 of 39 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning18 Apr
  2. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  4. 2025What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret3 Mar
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  6. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 10

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2025Group Sequence Policy Optimizationcited by 835
  7. 20242 OLMo 2 Furiouscited by 340
  8. 2025ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learningcited by 161
  9. 2025KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Codingcited by 119
  10. 2025Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoningcited by 68

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Learning to Correct: Calibrated Reinforcement Learning for Multi-Attempt Chain-of-Thoughtcited by 2
  2. 2026Skip-Connected Policy Optimization for Implicit Advantagecited by 1
  3. 2026TaPR: Test-Aware Policy Refinement for Feedback-Conditioned Code Generationcited by 1
  4. 2026Explore with Memory, Learn without It: In-Context Exploration and Self-Distillation for LLM Reasoningcited by 0
  5. 2026C3: Rethinking Process Supervision via Confidence-Guided Credit Correction in Multi-Turn Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Multi-Turn Code Generation Through Single-Step Rewards
  2. 2026DiDPO: Diff-in-Diff Policy Optimization for Coding Agent Training
  3. 2026TCPO: Turn-Level Credit Policy Optimization
  4. 2025From Reasoning to Code: GRPO Optimization for Underrepresented Languages
  5. 2025Empowering Multi-Turn Tool-Integrated Agentic Reasoning with Group Turn Policy Optimization
  6. 2026GRPO-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoning
  7. 2026Execution-Grounded Credit Assignment for GRPO in Code Generation
  8. 2026VETTA: Coordinating Turn- and Token-Level Credit Assignment for Multi-Turn LLM Agents