# From Evaluation to Optimisation: Hierarchy-Aware Training Signals for CWE Prediction in Python

arXiv: https://arxiv.org/abs/2607.21069 · HTML: https://mlsanity.com/paper/2607.21069/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 42 references that are in the map.
- First posted on arXiv 23 Jul 2026.

## Builds on (most cited first, 30 of 32)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021)
- [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://mlsanity.com/paper/2207.01780/) (2022)
- [When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs](https://mlsanity.com/paper/2406.01297/) (2024)
- [A Study of BFLOAT16 for Deep Learning Training](https://mlsanity.com/paper/1905.12322/) (2019)
- [MapCoder: Multi-Agent Code Generation for Competitive Problem Solving](https://mlsanity.com/paper/2405.11403/) (2024)
- [Large Language Models for Code: Security Hardening and Adversarial Testing](https://mlsanity.com/paper/2302.05319/) (2023)
- [RLTF: Reinforcement Learning from Unit Test Feedback](https://mlsanity.com/paper/2307.04349/) (2023)
- [Making Better Mistakes: Leveraging Class Hierarchies with Deep Networks](https://mlsanity.com/paper/1912.09393/) (2019)
- [Execution-based Code Generation using Deep Reinforcement Learning](https://mlsanity.com/paper/2301.13816/) (2023)
- [CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Software](https://mlsanity.com/paper/2107.08760/) (2021)
- [Instruction Tuning for Secure Code Generation](https://mlsanity.com/paper/2402.09497/) (2024)
- [How secure is AI-generated Code: A Large-Scale Comparison of Large Language Models](https://mlsanity.com/paper/2404.18353/) (2024)
- [Comparison of Static Application Security Testing Tools and Large Language Models for Repo-level Vulnerability Detection](https://mlsanity.com/paper/2407.16235/) (2024)
- [LLM-Powered Code Vulnerability Repair with Reinforcement Learning and Semantic Reward](https://mlsanity.com/paper/2401.03374/) (2024)
- [DLAP: A Deep Learning Augmented Large Language Model Prompting Framework for Software Vulnerability Detection](https://mlsanity.com/paper/2405.01202/) (2024)
- [Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation](https://mlsanity.com/paper/2506.23034/) (2025)
- [Code Security Vulnerability Repair Using Reinforcement Learning with Large Language Models](https://mlsanity.com/paper/2401.07031/) (2024)
- [Large Language Model Guided Self-Debugging Code Generation](https://mlsanity.com/paper/2502.02928/) (2025)
- [Pool Me Wisely: On the Effect of Pooling in Transformer-Based Models](https://mlsanity.com/paper/2510.03339/) (2025)
- [Secure Code Generation via Online Reinforcement Learning with Vulnerability Reward Model](https://mlsanity.com/paper/2602.07422/) (2026)
- [The Debugging Decay Index: Rethinking Debugging Strategies for Code LLMs](https://mlsanity.com/paper/2506.18403/) (2025)
- [Unleashing Artificial Cognition: Integrating Multiple AI Systems](https://mlsanity.com/paper/2408.04910/) (2024)
- [R+R: Security Vulnerability Dataset Quality Is Critical](https://mlsanity.com/paper/2503.06387/) (2025)

## Abstract (arXiv)

The original ALPHA benchmark introduced a taxonomy-aware penalty for evaluating CWE-level vulnerability prediction in Python and proposed that the penalty could theoretically also serve as a training signal. This paper provides that validation. We compare three delivery mechanisms: supervised fine-tuning, a dual-head classification loss, and reinforcement learning with a dense reward derived from the normalised penalty. We find that supervised approaches consistently regress below the zero-shot baseline under distribution shift, while GRPO succeeds. Our best policy reduces the cumulative ALPHA penalty of Qwen2.5-Coder-7B on Security Hardening and Adversarial Testing (SVEN) dataset by 27.9% under greedy decoding, and by 25.5% under sampled decoding(p = 0.005, Welch's t-test), reaching statistical parity with its 4.5x larger zero-shot teacher. We conclude that the value of a hierarchical penalty as a training signal depends largely on the directness of its delivery.