# Training Language Models to Generate Quality Code with Program Analysis Feedback

arXiv: https://arxiv.org/abs/2505.22704 · HTML: https://mlsanity.com/paper/2505.22704/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 37 references that are in the map.
- First posted on arXiv 28 May 2025.

## Builds on (most cited first, 16 of 16)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://mlsanity.com/paper/2207.01780/) (2022)
- [Large Language Models for Code: Security Hardening and Adversarial Testing](https://mlsanity.com/paper/2302.05319/) (2023)
- [Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models](https://mlsanity.com/paper/2312.04724/) (2023)
- [StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback](https://mlsanity.com/paper/2402.01391/) (2024)
- [Instruction Tuning for Secure Code Generation](https://mlsanity.com/paper/2402.09497/) (2024)
- [Teaching Language Models to Critique via Reinforcement Learning](https://mlsanity.com/paper/2502.03492/) (2025)
- [Constrained Decoding for Secure Code Generation](https://mlsanity.com/paper/2405.00218/) (2024)
- [PromSec: Prompt Optimization for Secure Generation of Functional Source Code with Large Language Models (LLMs)](https://mlsanity.com/paper/2409.12699/) (2024)
- [B-Coder: Value-Based Deep Reinforcement Learning for Program Synthesis](https://mlsanity.com/paper/2310.03173/) (2023)
- [SecCoder: Towards Generalizable and Robust Secure Code Generation](https://mlsanity.com/paper/2410.01488/) (2024)
- [Code Security Vulnerability Repair Using Reinforcement Learning with Large Language Models](https://mlsanity.com/paper/2401.07031/) (2024)

## Built on by (most cited first, 12 of 12)

- [A Survey of Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2509.08827/) (2025)
- [Learning to Optimize Multi-Objective Alignment Through Dynamic Reward Weighting](https://mlsanity.com/paper/2509.11452/) (2025)
- [A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic System](https://mlsanity.com/paper/2510.09721/) (2025)
- [Controlled Self-Evolution for Algorithmic Code Optimization](https://mlsanity.com/paper/2601.07348/) (2026)
- [From SFT to RL: Demystifying the Post-Training Pipeline for LLM-based Vulnerability Detection](https://mlsanity.com/paper/2602.14012/) (2026)
- [Uncovering Cross-Objective Interference in Multi-Objective Alignment](https://mlsanity.com/paper/2602.06869/) (2026)
- [Learning to Generate Secure Code via Token-Level Rewards](https://mlsanity.com/paper/2602.23407/) (2026)
- [Towards Pareto-Optimal Tool-Integrated Agents with Pareto Ranking Policy Optimization](https://mlsanity.com/paper/2606.16111/) (2026)
- [Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Code](https://mlsanity.com/paper/2605.05267/) (2026)
- [Position: Vibe Coding Needs Vibe Reasoning: Improving Vibe Coding with Formal Verification](https://mlsanity.com/paper/2511.00202/) (2025)
- [Characterizing Overconfident Failure in LLM-Based Code Generation](https://mlsanity.com/paper/2610.11300/) (2026)
- [LintEval: Benchmarking LLMs on Linter-Guided File Refactoring](https://mlsanity.com/paper/2610.20027/) (2026)

## Abstract (arXiv)

Code generation with large language models (LLMs), often termed vibe coding, is increasingly adopted in production but fails to ensure code quality, particularly in security (e.g., SQL injection vulnerabilities) and maintainability (e.g., missing type annotations). Existing methods, such as supervised fine-tuning and rule-based post-processing, rely on labor-intensive annotations or brittle heuristics, limiting their scalability and effectiveness. We propose REAL, a reinforcement learning framework that incentivizes LLMs to generate production-quality code using program analysis-guided feedback. Specifically, REAL integrates two automated signals: (1) program analysis detecting security or maintainability defects and (2) unit tests ensuring functional correctness. Unlike prior work, our framework is prompt-agnostic and reference-free, enabling scalable supervision without manual intervention. Experiments across multiple datasets and model scales demonstrate that REAL outperforms state-of-the-art methods in simultaneous assessments of functionality and code quality. Our work bridges the gap between rapid prototyping and production-ready code, enabling LLMs to deliver both speed and quality.