mlsanityData to 7 Oct 2026

Efficient Post-training of LLMs for Code Generation With Offline Reinforcement Learning

arXiv 2605.28409 ↗First posted 27 May 2026cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Post-training using online reinforcement learning (RL) is an important training step for LLMs, including code-generating models. However, online RL for code generation involves LLM inference and verification of the generated output, which can take considerable time and resources. In this paper, we explore the application of offline RL to code-generating models by leveraging existing code datasets. Our experiments demonstrate that offline RL is an effective training strategy for improving LLM performance. We show that offline RL can be especially beneficial for small LLMs and challenging coding problems.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 10 of its 14 references that are in the map. First posted on arXiv 27 May 2026.

Builds on

10 of 14 references.

References · 10

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  6. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  7. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  8. 2024A Survey on Large Language Models for Code Generationcited by 646
  9. 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 199
  10. 2024StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedbackcited by 74

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Performance, Efficiency and Collapse -- Advantages and Challenges in Offline Post-training of Code LLMs
  2. 2026Bridging Online and Offline RL: Contextual Bandit Learning for Multi-Turn Code Generation
  3. 2024Reinforcement Learning for LLM Post-Training: A Survey
  4. 2025Bridging Offline and Online Reinforcement Learning for LLMs
  5. 2025Exploring the Potential of Offline RL for Reasoning in LLMs: A Preliminary Study
  6. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
  7. 2024Applying RLAIF for Code Generation with API-usage in Lightweight LLMs
  8. 2025Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning