# CodeRL+: Improving Code Generation via Reinforcement with Execution Semantics Alignment

arXiv: https://arxiv.org/abs/2510.18471 · HTML: https://mlsanity.com/paper/2510.18471/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 54 references that are in the map.
- First posted on arXiv 21 Oct 2025.

## Builds on (most cited first, 30 of 40)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [Process Reinforcement through Implicit Rewards](https://mlsanity.com/paper/2502.01456/) (2025)
- [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://mlsanity.com/paper/2207.01780/) (2022)
- [A Survey on Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2402.13116/) (2024)
- [The False Promise of Imitating Proprietary LLMs](https://mlsanity.com/paper/2305.15717/) (2023)
- [LiveBench: A Challenging, Contamination-Limited LLM Benchmark](https://mlsanity.com/paper/2406.19314/) (2024)
- [Self-collaboration Code Generation via ChatGPT](https://mlsanity.com/paper/2304.07590/) (2023)
- [Skywork Open Reasoner 1 Technical Report](https://mlsanity.com/paper/2505.22312/) (2025)
- [TACO: Topics in Algorithmic COde generation dataset](https://mlsanity.com/paper/2312.14852/) (2023)
- [Teaching Large Language Models to Reason with Reinforcement Learning](https://mlsanity.com/paper/2403.04642/) (2024)
- [Predicting What You Already Know Helps: Provable Self-Supervised Learning](https://mlsanity.com/paper/2008.01064/) (2020)
- [Self-planning Code Generation with Large Language Models](https://mlsanity.com/paper/2303.06689/) (2023)
- [Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models](https://mlsanity.com/paper/2402.15938/) (2024)
- [OpenCodeReasoning: Advancing Data Distillation for Competitive Coding](https://mlsanity.com/paper/2504.01943/) (2025)
- [A Survey on Code Generation with LLM-based Agents](https://mlsanity.com/paper/2508.00083/) (2025)
- [LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs](https://mlsanity.com/paper/2504.14655/) (2025)

## Built on by (most cited first, 24 of 24)

- [Code as Agent Harness](https://mlsanity.com/paper/2605.18747/) (2026)
- [Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learning](https://mlsanity.com/paper/2605.26684/) (2026)
- [Controlled Self-Evolution for Algorithmic Code Optimization](https://mlsanity.com/paper/2601.07348/) (2026)
- [Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Tasks](https://mlsanity.com/paper/2604.02795/) (2026)
- [Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills](https://mlsanity.com/paper/2607.22529/) (2026)
- [CodeScaler: Scaling Code LLM Training and Test-Time Inference via Reward Models](https://mlsanity.com/paper/2602.17684/) (2026)
- [Execution-Grounded Credit Assignment for GRPO in Code Generation](https://mlsanity.com/paper/2603.16158/) (2026)
- [Self-Improvement of Large Language Models: A Technical Overview and Future Outlook](https://mlsanity.com/paper/2603.25681/) (2026)
- [CoRe-Code: Collaborative Reinforcement Learning for Code Generation](https://mlsanity.com/paper/2605.24812/) (2026)
- [LAD: Learning Advantage Distribution for Reasoning](https://mlsanity.com/paper/2602.20132/) (2026)
- [SPS: Steering Probability Squeezing for Better Exploration in Reinforcement Learning for Large Language Models](https://mlsanity.com/paper/2604.16995/) (2026)
- [Harnessing Routing Foresight for Micro-step-level MoE load balancing in RL Post-training](https://mlsanity.com/paper/2606.11867/) (2026)
- [Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignment](https://mlsanity.com/paper/2607.04728/) (2026)
- [Improving Small Language Models for Code Generation with Reinforcement Learning from Verification Feedback](https://mlsanity.com/paper/2605.30478/) (2026)
- [DiDPO: Diff-in-Diff Policy Optimization for Coding Agent Training](https://mlsanity.com/paper/2608.07147/) (2026)
- [Context Bootstrapped Reinforcement Learning](https://mlsanity.com/paper/2603.18953/) (2026)
- [Schedule-and-Calibrate: Utility-Guided Multi-Task Reinforcement Learning for Code LLMs](https://mlsanity.com/paper/2605.06111/) (2026)
- [Boosting Reinforcement Learning with Verifiable Rewards via Randomly Selected Few-Shot Guidance](https://mlsanity.com/paper/2605.15012/) (2026)
- [ConSteer-RL: Steering Reasoning Capabilities in Large Language Models via Confidence-Aware Reinforcement Learning](https://mlsanity.com/paper/2606.08088/) (2026)
- [RLPF: Reinforcement Learning from Performance Feedback for Code Generation](https://mlsanity.com/paper/2607.27271/) (2026)
- [AI Control Scientist: LLM-driven Agentic System for Automated Control Design](https://mlsanity.com/paper/2608.26780/) (2026)
- [Improving Large Language Models for Code through Runtime Program-State Reasoning](https://mlsanity.com/paper/2609.34359/) (2026)
- [ReMCTS: Reflection-Enhanced Monte Carlo Tree Search for Code Generation](https://mlsanity.com/paper/2609.34717/) (2026)
- [VIGOR: Verifiable Industrial Geometry Optimization and Repair in Scene Programs](https://mlsanity.com/paper/2610.13145/) (2026)

## Abstract (arXiv)

While Large Language Models (LLMs) excel at code generation by learning from vast code corpora, a fundamental semantic gap remains between their training on textual patterns and the goal of functional correctness, which is governed by formal execution semantics. Reinforcement Learning with Verifiable Rewards (RLVR) approaches attempt to bridge this gap using outcome rewards from executing test cases. However, solely relying on binary pass/fail signals is inefficient for establishing a well-aligned connection between the textual representation of code and its execution semantics, especially for subtle logical errors within the code. In this paper, we propose CodeRL+, a novel approach that integrates execution semantics alignment into the RLVR training pipeline for code generation. CodeRL+ enables the model to infer variable-level execution trajectory, providing a direct learning signal of execution semantics. CodeRL+ can construct execution semantics alignment directly using existing on-policy rollouts and integrates seamlessly with various RL algorithms. Extensive experiments demonstrate that CodeRL+ outperforms post-training baselines (including RLVR and Distillation), achieving a 4.6% average relative improvement in pass@1. CodeRL+ generalizes effectively to other coding tasks, yielding 15.5% and 4.4% higher accuracy on code-reasoning and test-output-generation benchmarks, respectively. CodeRL+ shows strong applicability across diverse RL algorithms and LLMs. Furthermore, probe analyses provide compelling evidence that CodeRL+ strengthens the alignment between code's textual representations and its underlying execution semantics.