mlsanityData to 7 Oct 2026

Integrating Symbolic Execution into the Fine-Tuning of Code-Generating LLMs

arXiv 2504.15210 ↗First posted 21 Apr 2025cs.SE cs.AIcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

Code-generating Large Language Models (LLMs) have become essential tools in modern software development, enhancing productivity and accelerating development. This paper aims to investigate the fine-tuning of code-generating LLMs using Reinforcement Learning and Direct Preference Optimization, further improving their performance. To achieve this, we enhance the training data for the reward model with the help of symbolic execution techniques, ensuring more comprehensive and objective data. With symbolic execution, we create a custom dataset that better captures the nuances in code evaluation. Our reward models, fine-tuned on this dataset, demonstrate significant improvements over the baseline, CodeRL, in estimating the quality of generated code. Our code-generating LLMs, trained with the help of reward model feedback, achieve similar results compared to the CodeRL benchmark.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 13 of its 21 references that are in the map. First posted on arXiv 21 Apr 2025.

Builds on

13 of 21 references.

References · 13

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  6. 2021Measuring Coding Challenge Competence With APPScited by 879
  7. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  8. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  9. 2023RLTF: Reinforcement Learning from Unit Test Feedbackcited by 110
  10. 2024StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedbackcited by 74
  11. 2022Compilable Neural Code Generation with Compiler Feedbackcited by 46
  12. 2023B-Coder: Value-Based Deep Reinforcement Learning for Program Synthesiscited by 16
1 more
  1. 2024Python Symbolic Execution with LLM-powered Code Generationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Teaching LLMs Program Semantics via Symbolic Execution Tracescited by 0
  2. 2026Learn from the Gap: Differential-Aware Advantage Pruning with Adaptive Rollout Sampling for GRPOcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Performance-Aligned LLMs for Generating Fast Code
  2. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
  3. 2024Process Supervision-Guided Policy Optimization for Code Generation
  4. 2026Domain-Adaptable Reinforcement Learning for Code Generation with Dense Rewards
  5. 2025Rethinking Verification for LLM Code Generation: From Generation to Testing
  6. 2026CodeScaler: Scaling Code LLM Training and Test-Time Inference via Reward Models
  7. 2025CodeRL+: Improving Code Generation via Reinforcement with Execution Semantics Alignment
  8. 2025Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimization