# Learning to Generate Secure Code via Token-Level Rewards

arXiv: https://arxiv.org/abs/2602.23407 · HTML: https://mlsanity.com/paper/2602.23407/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 68 references that are in the map.
- First posted on arXiv 26 Feb 2026.

## Builds on (most cited first, 25 of 25)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [Large Language Models for Code: Security Hardening and Adversarial Testing](https://mlsanity.com/paper/2302.05319/) (2023)
- [Vulnerability Detection with Code Language Models: How Far Are We?](https://mlsanity.com/paper/2403.18624/) (2024)
- [RLTF: Reinforcement Learning from Unit Test Feedback](https://mlsanity.com/paper/2307.04349/) (2023)
- [Instruction Tuning for Secure Code Generation](https://mlsanity.com/paper/2402.09497/) (2024)
- [CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generation](https://mlsanity.com/paper/2501.08200/) (2025)
- [CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Models](https://mlsanity.com/paper/2302.04012/) (2023)
- [HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation](https://mlsanity.com/paper/2412.21199/) (2024)
- [ProSec: Fortifying Code LLMs with Proactive Security Alignment](https://mlsanity.com/paper/2411.12882/) (2024)
- [ReCode: Reinforcing Code Generation with Reasoning-Process Rewards](https://mlsanity.com/paper/2508.05170/) (2025)
- [Large Language Model Supply Chain: A Research Agenda](https://mlsanity.com/paper/2404.12736/) (2024)
- [PurpCode: Reasoning for Safer Code Generation](https://mlsanity.com/paper/2507.19060/) (2025)
- [HexaCoder: Secure Code Generation via Oracle-Guided Synthetic Training Data](https://mlsanity.com/paper/2409.06446/) (2024)
- [Enhancing Large Language Models for Secure Code Generation: A Dataset-driven Study on Vulnerability Mitigation](https://mlsanity.com/paper/2310.16263/) (2023)
- [Training Language Models to Generate Quality Code with Program Analysis Feedback](https://mlsanity.com/paper/2505.22704/) (2025)
- [Teaching an Old LLM Secure Coding: Localized Preference Optimization on Distilled Preferences](https://mlsanity.com/paper/2506.00419/) (2025)
- [LLMs' Reshaping of People, Processes, Products, and Society in Software Development: A Comprehensive Exploration with Early Adopters](https://mlsanity.com/paper/2503.05012/) (2025)
- [Unseen Horizons: Unveiling the Real Capability of LLM Code Generation Beyond the Familiar](https://mlsanity.com/paper/2412.08109/) (2024)

## Built on by (most cited first, 2 of 2)

- [Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring](https://mlsanity.com/paper/2605.00754/) (2026)
- [SecureVibe: Making Vibe Coding More Secure](https://mlsanity.com/paper/2609.38606/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated strong capabilities in code generation, yet they remain prone to producing security vulnerabilities. Existing approaches commonly suffer from two key limitations: the scarcity of high-quality security data and coarse-grained reinforcement learning reward signals. To address these challenges, we propose Vul2Safe, a new secure code generation framework that leverages LLM self-reflection to construct high-confidence repair pairs from real-world vulnerabilities, and further generates diverse implicit prompts to build the PrimeVul+ dataset. Meanwhile, we introduce SRCode, a novel training framework that pioneers the use of token-level rewards in reinforcement learning for code security, which enables the model to continuously attend to and reinforce critical fine-grained security patterns during training. Compared with traditional instance-level reward schemes, our approach allows for more precise optimization of local security implementations. Extensive experiments show that PrimeVul+ and SRCode substantially reduce security vulnerabilities in generated code while improving overall code quality across multiple benchmarks.