# Multi-Turn Code Generation Through Single-Step Rewards

arXiv: https://arxiv.org/abs/2502.20380 · HTML: https://mlsanity.com/paper/2502.20380/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 37 of its 46 references that are in the map.
- First posted on arXiv 27 Feb 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Process reward models for reasoning](https://mlsanity.com/alignment/process-reward-models-for-reasoning/)

## Builds on (most cited first, 30 of 37)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://mlsanity.com/paper/2407.21787/) (2024)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [STaR: Bootstrapping Reasoning With Reasoning](https://mlsanity.com/paper/2203.14465/) (2022)
- [Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations](https://mlsanity.com/paper/2312.08935/) (2023)
- [Executable Code Actions Elicit Better LLM Agents](https://mlsanity.com/paper/2402.01030/) (2024)
- [Training Language Models to Self-Correct via Reinforcement Learning](https://mlsanity.com/paper/2409.12917/) (2024)
- [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://mlsanity.com/paper/2207.01780/) (2022)
- [ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search](https://mlsanity.com/paper/2406.03816/) (2024)
- [rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking](https://mlsanity.com/paper/2501.04519/) (2025)
- [Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning](https://mlsanity.com/paper/2410.08146/) (2024)
- [Generating Sequences by Learning to Self-Correct](https://mlsanity.com/paper/2211.00053/) (2022)
- [FireAct: Toward Language Agent Fine-tuning](https://mlsanity.com/paper/2310.05915/) (2023)
- [Thinking Fast and Slow with Deep Learning and Tree Search](https://mlsanity.com/paper/1705.08439/) (2017)
- [ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL](https://mlsanity.com/paper/2402.19446/) (2024)
- [Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning](https://mlsanity.com/paper/2302.02662/) (2023)
- [OctoPack: Instruction Tuning Code Large Language Models](https://mlsanity.com/paper/2308.07124/) (2023)
- [MapCoder: Multi-Agent Code Generation for Competitive Problem Solving](https://mlsanity.com/paper/2405.11403/) (2024)
- [Recursive Introspection: Teaching Language Model Agents How to Self-Improve](https://mlsanity.com/paper/2407.18219/) (2024)
- [Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning](https://mlsanity.com/paper/2405.10292/) (2024)
- [RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning](https://mlsanity.com/paper/2410.02089/) (2024)
- [Reinforcement and Imitation Learning via Interactive No-Regret Learning](https://mlsanity.com/paper/1406.5979/) (2014)

## Built on by (most cited first, 28 of 28)

- [Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks](https://mlsanity.com/paper/2602.22817/) (2026)
- [VisCoder: Fine-Tuning LLMs for Executable Python Visualization Code Generation](https://mlsanity.com/paper/2506.03930/) (2025)
- [A Smooth Sea Never Made a Skilled SAILOR: Robust Imitation via Learning to Search](https://mlsanity.com/paper/2506.05294/) (2025)
- [Calibrate-Then-Act: Cost-Aware Exploration in LLM Agents](https://mlsanity.com/paper/2602.16699/) (2026)
- [Generative AI Act II: Test Time Scaling Drives Cognition Engineering](https://mlsanity.com/paper/2504.13828/) (2025)
- [Scaling Agents for Computer Use](https://mlsanity.com/paper/2510.02250/) (2025)
- [V1: Unifying Generation and Self-Verification for Parallel Reasoners](https://mlsanity.com/paper/2603.04304/) (2026)
- [RedCoder: Automated Multi-Turn Red Teaming for Code LLMs](https://mlsanity.com/paper/2507.22063/) (2025)
- [Learning from Failures in Multi-Attempt Reinforcement Learning](https://mlsanity.com/paper/2503.04808/) (2025)
- [Agnostics: Learning to Code in Any Programming Language via Reinforcement with a Universal Learning Environment](https://mlsanity.com/paper/2508.04865/) (2025)
- [AgentPack: A Dataset of Code Changes, Co-Authored by Agents and Humans](https://mlsanity.com/paper/2509.21891/) (2025)
- [MM-ReCoder: Advancing Chart-to-Code Generation with Reinforcement Learning and Self-Correction](https://mlsanity.com/paper/2604.01600/) (2026)
- [Self-Correcting Code Generation Using Small Language Models](https://mlsanity.com/paper/2505.23060/) (2025)
- [Bridging Online and Offline RL: Contextual Bandit Learning for Multi-Turn Code Generation](https://mlsanity.com/paper/2602.03806/) (2026)
- [Generalizable LLM Learning of Graph Synthetic Data with Post-training Alignment](https://mlsanity.com/paper/2506.00845/) (2025)
- [RM -RF: Reward Model for Run-Free Unit Test Evaluation](https://mlsanity.com/paper/2601.13097/) (2026)
- [RefineRL: Advancing Competitive Programming with Self-Refinement Reinforcement Learning](https://mlsanity.com/paper/2604.00790/) (2026)
- [Credit-Budgeted ICPC-Style Coding: When Agents Must Pay for Every Decision](https://mlsanity.com/paper/2604.10182/) (2026)
- [Learning to Correct: Calibrated Reinforcement Learning for Multi-Attempt Chain-of-Thought](https://mlsanity.com/paper/2604.17912/) (2026)
- [Autoregressive, Yet Revisable: In Decoding Revision for Secure Code Generation](https://mlsanity.com/paper/2602.01187/) (2026)
- [Learning the ARTS of Search for Automated Discovery](https://mlsanity.com/paper/2606.21891/) (2026)
- [TaPR: Test-Aware Policy Refinement for Feedback-Conditioned Code Generation](https://mlsanity.com/paper/2608.00494/) (2026)
- [PIPer: On-Device Environment Setup via Online Reinforcement Learning](https://mlsanity.com/paper/2509.25455/) (2025)
- [BoostLoRA: Growing Effective Rank by Boosting Adapters](https://mlsanity.com/paper/2604.27308/) (2026)
- [Beyond Execution: Static-Analysis Rewards and Hint-Conditioned Diffusion RL for Code Generation](https://mlsanity.com/paper/2605.17174/) (2026)
- [REVES: REvision and VErification--Augmented Training for Test-Time Scaling](https://mlsanity.com/paper/2606.18910/) (2026)
- [LEAP: Lean Environment-Feedback via Adaptive Pruning for Code RL in GPU Kernel Generation](https://mlsanity.com/paper/2608.01804/) (2026)
- [Explore with Memory, Learn without It: In-Context Exploration and Self-Distillation for LLM Reasoning](https://mlsanity.com/paper/2610.17734/) (2026)

## Abstract (arXiv)

We address the problem of code generation from multi-turn execution feedback. Existing methods either generate code without feedback or use complex, hierarchical reinforcement learning to optimize multi-turn rewards. We propose a simple yet scalable approach, μCode, that solves multi-turn code generation using only single-step rewards. Our key insight is that code generation is a one-step recoverable MDP, where the correct code can be recovered from any intermediate code state in a single turn. μCode iteratively trains both a generator to provide code solutions conditioned on multi-turn execution feedback and a verifier to score the newly generated code. Experimental evaluations show that our approach achieves significant improvements over the state-of-the-art baselines. We provide analysis of the design choices of the reward models and policy, and show the efficacy of μCode at utilizing the execution feedback. Our code is available at https://github.com/portal-cornell/muCode.