mlsanityData to 14 Oct 2026

Multi-Turn Code Generation Through Single-Step Rewards

arXiv 2502.20380 ↗First posted 27 Feb 2025cs.LG cs.AI cs.CLcited by 28

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

We address the problem of code generation from multi-turn execution feedback. Existing methods either generate code without feedback or use complex, hierarchical reinforcement learning to optimize multi-turn rewards. We propose a simple yet scalable approach, μCode, that solves multi-turn code generation using only single-step rewards. Our key insight is that code generation is a one-step recoverable MDP, where the correct code can be recovered from any intermediate code state in a single turn. μCode iteratively trains both a generator to provide code solutions conditioned on multi-turn execution feedback and a verifier to score the newly generated code. Experimental evaluations show that our approach achieves significant improvements over the state-of-the-art baselines. We provide analysis of the design choices of the reward models and policy, and show the efficacy of μCode at utilizing the execution feedback. Our code is available at https://github.com/portal-cornell/muCode.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 37 of its 46 references that are in the map. First posted on arXiv 27 Feb 2025.

Builds on

37 of 46 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning10 Oct
  2. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search6 Jun
  3. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec

Other references · 34

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  8. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  9. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  10. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  11. 2023Teaching Large Language Models to Self-Debugcited by 988
  12. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
22 more
  1. 2024Executable Code Actions Elicit Better LLM Agentscited by 564
  2. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  3. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 358
  4. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 313
  5. 2022Generating Sequences by Learning to Self-Correctcited by 277
  6. 2023FireAct: Toward Language Agent Fine-tuningcited by 256
  7. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 252
  8. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 228
  9. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 195
  10. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 189
  11. 2024MapCoder: Multi-Agent Code Generation for Competitive Problem Solvingcited by 177
  12. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 165
  13. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  14. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 151
  15. 2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 118
  16. 2024Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineeringcited by 103
  17. 2023Execution-based Code Generation using Deep Reinforcement Learningcited by 99
  18. 2017Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Predictioncited by 93
  19. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gapcited by 81
  20. 2024NExT: Teaching Large Language Models to Reason about Code Executioncited by 63
  21. 2024Commit0: Library Generation from Scratchcited by 52
  22. 2024Better than Your Teacher: LLM Agents that learn from Privileged AI Feedbackcited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 17.

28 papers.

Sort:

By area:

  1. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Taskscited by 49
  2. 2025VisCoder: Fine-Tuning LLMs for Executable Python Visualization Code Generationcited by 17
  3. 2025A Smooth Sea Never Made a Skilled SAILOR: Robust Imitation via Learning to Searchcited by 17
  4. 2026Calibrate-Then-Act: Cost-Aware Exploration in LLM Agentscited by 17
  5. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 15
  6. 2025Scaling Agents for Computer Usecited by 14
  7. 2026V1: Unifying Generation and Self-Verification for Parallel Reasonerscited by 12
  8. 2025RedCoder: Automated Multi-Turn Red Teaming for Code LLMscited by 9
  9. 2025Learning from Failures in Multi-Attempt Reinforcement Learningcited by 6
  10. 2025Agnostics: Learning to Code in Any Programming Language via Reinforcement with a Universal Learning Environmentcited by 5
  11. 2025AgentPack: A Dataset of Code Changes, Co-Authored by Agents and Humanscited by 5
  12. 2026MM-ReCoder: Advancing Chart-to-Code Generation with Reinforcement Learning and Self-Correctioncited by 5
16 more
  1. 2025Self-Correcting Code Generation Using Small Language Modelscited by 4
  2. 2026Bridging Online and Offline RL: Contextual Bandit Learning for Multi-Turn Code Generationcited by 4
  3. 2025Generalizable LLM Learning of Graph Synthetic Data with Post-training Alignmentcited by 3
  4. 2026RM -RF: Reward Model for Run-Free Unit Test Evaluationcited by 2
  5. 2026RefineRL: Advancing Competitive Programming with Self-Refinement Reinforcement Learningcited by 2
  6. 2026Credit-Budgeted ICPC-Style Coding: When Agents Must Pay for Every Decisioncited by 2
  7. 2026Learning to Correct: Calibrated Reinforcement Learning for Multi-Attempt Chain-of-Thoughtcited by 2
  8. 2026Autoregressive, Yet Revisable: In Decoding Revision for Secure Code Generationcited by 1
  9. 2026Learning the ARTS of Search for Automated Discoverycited by 1
  10. 2026TaPR: Test-Aware Policy Refinement for Feedback-Conditioned Code Generationcited by 1
  11. 2025PIPer: On-Device Environment Setup via Online Reinforcement Learningcited by 0
  12. 2026BoostLoRA: Growing Effective Rank by Boosting Adapterscited by 0
  13. 2026Beyond Execution: Static-Analysis Rewards and Hint-Conditioned Diffusion RL for Code Generationcited by 0
  14. 2026REVES: REvision and VErification--Augmented Training for Test-Time Scalingcited by 0
  15. 2026LEAP: Lean Environment-Feedback via Adaptive Pruning for Code RL in GPU Kernel Generationcited by 0
  16. 2026Explore with Memory, Learn without It: In-Context Exploration and Self-Distillation for LLM Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MURPHY: Feedback-Aware GRPO with Retrospective Credit Assignment for Multi-Turn Code Generation
  2. 2026Exploring Pass-Rate Reward in Reinforcement Learning for Code Generation
  3. 2026RLPF: Reinforcement Learning from Performance Feedback for Code Generation
  4. 2024StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback
  5. 2026Domain-Adaptable Reinforcement Learning for Code Generation with Dense Rewards
  6. 2024Process Supervision-Guided Policy Optimization for Code Generation
  7. 2026MaxCode: A Max-Reward Reinforcement Learning Framework for Automated Code Optimization
  8. 2025More with Less: An Empirical Study of Turn-Control Strategies for Efficient Coding Agents