# EvalSVA: Multi-Agent Evaluators for Next-Gen Software Vulnerability Assessment

arXiv: https://arxiv.org/abs/2501.14737 · HTML: https://mlsanity.com/paper/2501.14737/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 11 of its 34 references that are in the map.
- First posted on arXiv 11 Dec 2024.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 11 of 11)

- [Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate](https://mlsanity.com/paper/2305.19118/) (2023)
- [What Makes Good In-context Demonstrations for Code Intelligence Tasks with LLMs?](https://mlsanity.com/paper/2304.07575/) (2023)
- [Multitask-based Evaluation of Open-Source LLM on Software Vulnerability](https://mlsanity.com/paper/2404.02056/) (2024)
- [SPI: Automated Identification of Security Patches via Commits](https://mlsanity.com/paper/2105.14565/) (2021)
- [A Survey on Data-driven Software Vulnerability Assessment and Prioritization](https://mlsanity.com/paper/2107.08364/) (2021)
- [AIBugHunter: A Practical Tool for Predicting, Classifying and Repairing Software Vulnerabilities](https://mlsanity.com/paper/2305.16615/) (2023)
- [Automated Software Vulnerability Assessment with Concept Drift](https://mlsanity.com/paper/2103.11316/) (2021)
- [DeepCVA: Automated Commit-level Vulnerability Assessment with Deep Multi-task Learning](https://mlsanity.com/paper/2108.08041/) (2021)
- [Uncovering the Causes of Emotions in Software Developer Communication Using Zero-shot LLMs](https://mlsanity.com/paper/2312.09731/) (2023)
- [Early Detection of Security-Relevant Bug Reports using Machine Learning: How Far Are We?](https://mlsanity.com/paper/2112.10123/) (2021)
- [An Empirical Study of Automation in Software Security Patch Management](https://mlsanity.com/paper/2209.01518/) (2022)

## Built on by (most cited first, 6 of 6)

- [Let the Trial Begin: A Mock-Court Approach to Vulnerability Detection using LLM-Based Agents](https://mlsanity.com/paper/2505.10961/) (2025)
- [VulAgent: Hypothesis-Validation based Multi-Agent Vulnerability Detection](https://mlsanity.com/paper/2509.11523/) (2025)
- [CodeVisionary: An Agent-based Framework for Evaluating Large Language Models in Code Generation](https://mlsanity.com/paper/2504.13472/) (2025)
- [MAVUL: Multi-Agent Vulnerability Detection via Contextual Reasoning and Interactive Refinement](https://mlsanity.com/paper/2510.00317/) (2025)
- [Multi-Agent Debate Strategies: Survey, Taxonomy, and Challenges](https://mlsanity.com/paper/2607.26212/) (2026)
- [Three Heads Are Better Than One: A Multi-perspective Reasoning Framework for Enhanced Vulnerability Detection](https://mlsanity.com/paper/2605.18153/) (2026)

## Abstract (arXiv)

Software Vulnerability (SV) assessment is a crucial process of determining different aspects of SVs (e.g., attack vectors and scope) for developers to effectively prioritize efforts in vulnerability mitigation. It presents a challenging and laborious process due to the complexity of SVs and the scarcity of labeled data. To mitigate the above challenges, we introduce EvalSVA, a multi-agent evaluators team to autonomously deliberate and evaluate various aspects of SV assessment. Specifically, we propose a multi-agent-based framework to simulate vulnerability assessment strategies in real-world scenarios, which employs multiple Large Language Models (LLMs) into an integrated group to enhance the effectiveness of SV assessment in the limited data. We also design diverse communication strategies to autonomously discuss and assess different aspects of SV. Furthermore, we construct a multi-lingual SV assessment dataset based on the new standard of CVSS, comprising 699, 888, and 1,310 vulnerability-related commits in C++, Python, and Java, respectively. Our experimental results demonstrate that EvalSVA averagely outperforms the 44.12% accuracy and 43.29% F1 for SV assessment compared with the previous methods. It shows that EvalSVA offers a human-like process and generates both reason and answer for SV assessment. EvalSVA can also aid human experts in SV assessment, which provides more explanation and details for SV assessment.