mlsanityData to 14 Oct 2026

Bridging Online and Offline RL: Contextual Bandit Learning for Multi-Turn Code Generation

arXiv 2602.03806 ↗First posted 3 Feb 2026cs.LG cs.AI cs.CLcited by 4

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Training large language models (LLMs) for multi-turn code generation typically relies on online reinforcement learning (RL), in which models learn from repeatedly generated online rollouts. Although effective, online RL can be costly and unstable. In this paper, we present contextual bandit learning with offline trajectories (COBALT), a simple method that reuses offline multi-turn trajectories to efficiently train LLMs for multi-turn code generation. Building on the observation that multi-turn code generation is a one-step recoverable Markov decision process, COBALT first collects code generation trajectories using a reference LLM and divides them into partial trajectories. Then, during online bandit learning, the LLM is trained to complete each partial trajectory through single-step code generation. On LiveCodeBench, COBALT substantially improves R1-Distill 8B and Qwen3 8B by up to 9.0 and 6.2 absolute Pass@1 points and matches the performance of two multi-turn online RL baselines using GRPO and VeRPO. We also perform a perturbation-based analysis of in-context reward hacking and show that Cobalt can improve robustness to misleading execution feedback. Overall, our results demonstrate that Cobalt is a promising solution for multi-turn code generation. Our code and data are available at https://github.com/OSU-NLP-Group/cobalt.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 30 of its 42 references that are in the map. First posted on arXiv 3 Feb 2026.

Builds on

30 of 42 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 29

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  7. 2023Teaching Large Language Models to Self-Debugcited by 988
  8. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  9. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  10. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
  11. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 358
  12. 2023Mitigating the Alignment Tax of RLHFcited by 183
17 more
  1. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 151
  2. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoningcited by 146
  3. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 136
  4. 2024Language Models Learn to Mislead Humans via RLHFcited by 124
  5. 2025Between Underthinking and Overthinking: An Empirical Study of Reasoning Length and correctness in LLMscited by 117
  6. 2023RLTF: Reinforcement Learning from Unit Test Feedbackcited by 114
  7. 2023Execution-based Code Generation using Deep Reinforcement Learningcited by 99
  8. 2023Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debatecited by 94
  9. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learningcited by 84
  10. 2024Feedback Loops With Language Models Drive In-Context Reward Hackingcited by 77
  11. 2025ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Casescited by 65
  12. 2025Does Thinking More always Help? Mirage of Test-Time Scaling in Reasoning Modelscited by 53
  13. 2025Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interactioncited by 34
  14. 2025Multi-Turn Code Generation Through Single-Step Rewardscited by 28
  15. 2024Spontaneous Reward Hacking in Iterative Self-Refinementcited by 26
  16. 2025ConvCodeWorld: Benchmarking Conversational Code Generation in Reproducible Feedback Environmentscited by 14
  17. 2024Honesty to Subterfuge: In-Context Reinforcement Learning Can Make Honest Models Reward Hackcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Beyond pass@k: Redundancy-Aware RLVR for Multi-Sample Code Generationcited by 1
  2. 2026TaPR: Test-Aware Policy Refinement for Feedback-Conditioned Code Generationcited by 1
  3. 2026PYTHALAB-MERA: Validation-Grounded Memory, Retrieval, and Acceptance Control for Frozen-LLM Coding Agentscited by 0
  4. 2026Strategically Diverse Sampling for Self-Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Efficient Post-training of LLMs for Code Generation With Offline Reinforcement Learning
  2. 2026Performance, Efficiency and Collapse -- Advantages and Challenges in Offline Post-training of Code LLMs
  3. 2024LLMs Are In-Context Bandit Reinforcement Learners
  4. 2026DRIFT: Decoupled Rollouts and Importance-Weighted Fine-Tuning for Efficient Multi-Turn Optimization
  5. 2024Enhancing Code LLMs with Reinforcement Learning in Code Generation: A Survey
  6. 2025Bridging Offline and Online Reinforcement Learning for LLMs
  7. 2023Semi-Offline Reinforcement Learning for Optimized Text Generation
  8. 2024Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Search