# Multitask-based Evaluation of Open-Source LLM on Software Vulnerability

arXiv: https://arxiv.org/abs/2404.02056 · HTML: https://mlsanity.com/paper/2404.02056/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Apr 2024.

## Built on by (most cited first, 23 of 23)

- [LLMs in Software Security: A Survey of Vulnerability Detection Techniques and Insights](https://mlsanity.com/paper/2502.07049/) (2025)
- [Rectifier: Code Translation with Corrector via LLMs](https://mlsanity.com/paper/2407.07472/) (2024)
- [How Well Do Large Language Models Serve as End-to-End Secure Code Agents for Python?](https://mlsanity.com/paper/2408.10495/) (2024)
- [EvalSVA: Multi-Agent Evaluators for Next-Gen Software Vulnerability Assessment](https://mlsanity.com/paper/2501.14737/) (2024)
- [One Bug, Hundreds Behind: LLMs for Large-Scale Bug Discovery](https://mlsanity.com/paper/2510.14036/) (2025)
- [Toward Intelligent and Secure Cloud: Large Language Model Empowered Proactive Defense](https://mlsanity.com/paper/2412.21051/) (2024)
- [From SFT to RL: Demystifying the Post-Training Pipeline for LLM-based Vulnerability Detection](https://mlsanity.com/paper/2602.14012/) (2026)
- [Revisiting Pre-trained Language Models for Vulnerability Detection](https://mlsanity.com/paper/2507.16887/) (2025)
- [Generative transformations and patterns in LLM-native approaches for software verification and falsification](https://mlsanity.com/paper/2404.09384/) (2024)
- [Adaptive Plan-Execute Framework for Smart Contract Security Auditing](https://mlsanity.com/paper/2505.15242/) (2025)
- [Towards Secure Logging: Characterizing and Benchmarking Logging Code Security Issues with LLMs](https://mlsanity.com/paper/2604.20211/) (2026)
- [Unveiling the Learning Mind of Language Models: A Cognitive Framework and Empirical Study](https://mlsanity.com/paper/2506.13464/) (2025)
- [CVE-LLM : Automatic vulnerability evaluation in medical device industry using large language models](https://mlsanity.com/paper/2407.14640/) (2024)
- [Breaking the Cycle of Recurring Failures: Applying Generative AI to Root Cause Analysis in Legacy Banking Systems](https://mlsanity.com/paper/2411.13017/) (2024)
- [ML For Hardware Design Interpretability: Challenges and Opportunities](https://mlsanity.com/paper/2504.08852/) (2025)
- [Security smells in infrastructure as code: a taxonomy update beyond the seven sins](https://mlsanity.com/paper/2509.18761/) (2025)
- [Detection of security smells in IaC scripts through semantics-aware code and language processing](https://mlsanity.com/paper/2509.18790/) (2025)
- [MulVuln: Enhancing Pre-trained LMs with Shared and Language-Specific Knowledge for Multilingual Vulnerability Detection](https://mlsanity.com/paper/2510.04397/) (2025)
- [SoK: Understanding (New) Security Issues Across AI4Code Use Cases](https://mlsanity.com/paper/2512.18456/) (2025)
- [On the Effectiveness of Instruction-Tuning Local LLMs for Identifying Software Vulnerabilities](https://mlsanity.com/paper/2512.20062/) (2025)
- [SecureRAG-RTL: A Retrieval-Augmented, Multi-Agent, Zero-Shot LLM-Driven Framework for Hardware Vulnerability Detection](https://mlsanity.com/paper/2603.05689/) (2026)
- [Conversations Risk Detection LLMs in Financial Agents via Multi-Stage Generative Rollout](https://mlsanity.com/paper/2604.09056/) (2026)
- [Data-Centric Benchmarking of Exploit Generation in LLMs: Understanding the Impact of Fine-Tuning](https://mlsanity.com/paper/2606.15123/) (2026)

## Abstract (arXiv)

This paper proposes a pipeline for quantitatively evaluating interactive Large Language Models (LLMs) using publicly available datasets. We carry out an extensive technical evaluation of LLMs using Big-Vul covering four different common software vulnerability tasks. This evaluation assesses the multi-tasking capabilities of LLMs based on this dataset. We find that the existing state-of-the-art approaches and pre-trained Language Models (LMs) are generally superior to LLMs in software vulnerability detection. However, in software vulnerability assessment and location, certain LLMs (e.g., CodeLlama and WizardCoder) have demonstrated superior performance compared to pre-trained LMs, and providing more contextual information can enhance the vulnerability assessment capabilities of LLMs. Moreover, LLMs exhibit strong vulnerability description capabilities, but their tendency to produce excessive output significantly weakens their performance compared to pre-trained LMs. Overall, though LLMs perform well in some aspects, they still need improvement in understanding the subtle differences in code vulnerabilities and the ability to describe vulnerabilities to fully realize their potential. Our evaluation pipeline provides valuable insights into the capabilities of LLMs in handling software vulnerabilities.