# Evaluating Large Language Models for Security Bug Report Prediction

arXiv: https://arxiv.org/abs/2601.22921 · HTML: https://mlsanity.com/paper/2601.22921/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 27 references that are in the map.
- First posted on arXiv 30 Jan 2026.

## Builds on (most cited first, 11 of 11)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Better Zero-Shot Reasoning with Role-Play Prompting](https://mlsanity.com/paper/2308.07702/) (2023)
- [HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation](https://mlsanity.com/paper/2412.21199/) (2024)
- [Distilled GPT for Source Code Summarization](https://mlsanity.com/paper/2308.14731/) (2023)
- [HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex Diagrams in Coding Tasks](https://mlsanity.com/paper/2410.12381/) (2024)
- [Benchmarking Prompt Engineering Techniques for Secure Code Generation with GPT Models](https://mlsanity.com/paper/2502.06039/) (2025)
- [How Do Developers Deal with Security Issue Reports on GitHub?](https://mlsanity.com/paper/2112.10359/) (2021)
- [Better Security Bug Report Classification via Hyperparameter Optimization](https://mlsanity.com/paper/1905.06872/) (2019)
- [Security Bug Report Prediction Within and Across Projects: A Comparative Study of BERT and Random Forest](https://mlsanity.com/paper/2504.21037/) (2025)

## Built on by (most cited first, 1 of 1)

- [From Backlog Items to Security Guidance: Towards Continuous Security Compliance](https://mlsanity.com/paper/2607.27374/) (2026)

## Abstract (arXiv)

Early detection of security bug reports (SBRs) is critical for timely vulnerability mitigation. We present an evaluation of prompt-based engineering and fine-tuning approaches for predicting SBRs using Large Language Models (LLMs). Our findings reveal a distinct trade-off between the two approaches. Prompted proprietary models demonstrate the highest sensitivity to SBRs, achieving a G-measure of 77% and a recall of 74% on average across all the datasets, albeit at the cost of a higher false-positive rate, resulting in an average precision of only 22%. Fine-tuned models, by contrast, exhibit the opposite behavior, attaining a lower overall G-measure of 51% but substantially higher precision of 75% at the cost of reduced recall of 36%. Though a one-time investment in building fine-tuned models is necessary, the inference on the largest dataset is up to 50 times faster than that of proprietary models. These findings suggest that further investigations to harness the power of LLMs for SBR prediction are necessary.