mlsanityData to 7 Oct 2026

Signal Reshaping for GRPO in Weak-Feedback Agentic Code Repair

arXiv 2605.07276 ↗First posted 8 May 2026cs.AIcited by 0

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Code-agent RL often receives weak feedback: rollout-time signals are reliable and executable, but capture only necessary or surface conditions for task success rather than the target semantic predicate. Using agentic compile-fix as the setting, we study signal reshaping for standard GRPO under such feedback. Our central claim is that GRPO's within-group comparison is meaningful only after three kinds of signals are reshaped: outcome rewards recover semantic ranking, process signals localize intra-trajectory credit, and rollouts from the same prompt remain execution-comparable. We operationalize these conditions with a minimal signal-reshaping construction that leaves GRPO's group-normalized advantage construction unchanged: compile-and-semantic layered rewards reshape trajectory ranking, step-level process scores outside group reward normalization reshape within-trajectory update strength, and failure-cause-aware rollout governance reshapes within-group comparability. Experiments show a clear end-to-end gain: full signal-reshaped GRPO improves strict compile-and-semantic accuracy from the base model's zero-shot 0.385 to 0.535. Controlled comparisons further explain the source of this gain: binary rewards remove the compile-only middle tier and degrade trajectory control; on top of layered rewards, process-score weighting further improves accuracy from 0.48 to 0.53 and reduces average evaluation steps from 23.50 to 17.02. As a boundary comparison, privileged-prompt token-level distillation mainly optimizes local distributional alignment; in long tool-use trajectories, this signal is diluted by non-critical tokens and cannot replace outcome semantics, process credit, or within-group comparability.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 47 of its 63 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

47 of 63 references.

Closest ancestors · 1

Same direction, newest first

  1. 2026From Laboratory to Real-World Applications: Benchmarking Agentic Code Reasoning at the Repository Level7 Jan

Other references · 46

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  6. 2023Let's Verify Step by Stepcited by 3,598
  7. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  9. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  10. 2016Concrete Problems in AI Safetycited by 1,582
  11. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  12. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
34 more
  1. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  2. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  3. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  4. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  5. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 426
  6. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
  7. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  8. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  9. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 246
  10. 2022Defining and Characterizing Reward Hackingcited by 184
  11. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimizationcited by 175
  12. 2025Multi-SWE-bench: A Multilingual Benchmark for Issue Resolvingcited by 165
  13. 2024Free Process Rewards without Process Labelscited by 162
  14. 2026A Survey of On-Policy Distillation for Large Language Modelscited by 148
  15. 2018SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repaircited by 144
  16. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 122
  17. 2023RLTF: Reinforcement Learning from Unit Test Feedbackcited by 110
  18. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplificationcited by 108
  19. 2023Execution-based Code Generation using Deep Reinforcement Learningcited by 95
  20. 2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoningcited by 94
  21. 2021CURE: Code-Aware Neural Machine Translation for Automatic Program Repaircited by 85
  22. 2026Privileged Information Distillation for Language Modelscited by 79
  23. 2026Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillationcited by 53
  24. 2023RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repaircited by 50
  25. 2025Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoningcited by 49
  26. 2025KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learningcited by 35
  27. 2025Kimi-Dev: Agentless Training as Skill Prior for SWE-Agentscited by 35
  28. 2025Training Long-Context, Multi-Turn Software Engineering Agents with Reinforcement Learningcited by 34
  29. 2026GATES: Self-Distillation under Privileged Context with Consensus Gatingcited by 30
  30. 2025MO-GRPO: Mitigating Reward Hacking of Group Relative Policy Optimization on Multi-Objective Problemscited by 21
  31. 2026RAGEN-2: Reasoning Collapse in Agentic RLcited by 16
  32. 2025Student-Centered Distillation Narrows the Agentic Gap Between Small and Large LLMscited by 11
  33. 2026RepoRepair: Leveraging Code Documentation for Repository-Level Automated Program Repaircited by 5
  34. 2026VulKey: Automated Vulnerability Repair Guided by Domain-Specific Repair Patternscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond Accuracy: Behavioral Dynamics of Agentic Multi-Hunk Repair
  2. 2026Coherence Collapse: Diagnosing Why Code Agents Fail After Reaching the Right Code
  3. 2025MURPHY: Feedback-Aware GRPO with Retrospective Credit Assignment for Multi-Turn Code Generation
  4. 2026TaPR: Test-Aware Policy Refinement for Feedback-Conditioned Code Generation
  5. 2026ProcCtrlBench: Evaluating Process-Level Defects and Control Preservation in LLM Coding Agents
  6. 2026PERFOPT-Bench: Evaluating Coding Agents on Software Performance Optimization
  7. 2026Repair the Amplifier, Not the Symptom: Stable World-Model Correction for Agent Rollouts
  8. 2026Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair