# From LLMs to Agents: A Comparative Evaluation of LLMs and LLM-based Agents in Security Patch Detection

arXiv: https://arxiv.org/abs/2511.08060 · HTML: https://mlsanity.com/paper/2511.08060/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 28 of its 70 references that are in the map.
- First posted on arXiv 11 Nov 2025.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 28 of 28)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Gated Graph Sequence Neural Networks](https://mlsanity.com/paper/1511.05493/) (2015)
- [LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead](https://mlsanity.com/paper/2404.04834/) (2024)
- [LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigation](https://mlsanity.com/paper/2409.20550/) (2024)
- [RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generation](https://mlsanity.com/paper/2402.16667/) (2024)
- [RAP-Gen: Retrieval-Augmented Patch Generation with CodeT5 for Automatic Program Repair](https://mlsanity.com/paper/2309.06057/) (2023)
- [Can Large Language Models Identify Authorship?](https://mlsanity.com/paper/2403.08213/) (2024)
- [CodeCoR: An LLM-Based Self-Reflective Multi-Agent Framework for Code Generation](https://mlsanity.com/paper/2501.07811/) (2025)
- [Benchmarking LLMs and LLM-based Agents in Practical Vulnerability Detection for Code Repositories](https://mlsanity.com/paper/2503.03586/) (2025)
- [Teaching Code LLMs to Use Autocompletion Tools in Repository-Level Code Generation](https://mlsanity.com/paper/2401.06391/) (2024)
- [MARE: Multi-Agents Collaboration Framework for Requirements Engineering](https://mlsanity.com/paper/2405.03256/) (2024)
- [SPI: Automated Identification of Security Patches via Commits](https://mlsanity.com/paper/2105.14565/) (2021)
- [DocAgent: A Multi-Agent System for Automated Code Documentation Generation](https://mlsanity.com/paper/2504.08725/) (2025)
- [PatchRNN: A Deep Learning-Based System for Security Patch Identification](https://mlsanity.com/paper/2108.03358/) (2021)
- [VulCurator: A Vulnerability-Fixing Commit Detector](https://mlsanity.com/paper/2209.03260/) (2022)
- [The Best of Both Worlds: Combining Learned Embeddings with Engineered Features for Accurate Prediction of Correct Patches](https://mlsanity.com/paper/2203.08912/) (2022)
- [VulAgent: Hypothesis-Validation based Multi-Agent Vulnerability Detection](https://mlsanity.com/paper/2509.11523/) (2025)
- [Enhancing Security Patch Identification by Capturing Structures in Commits](https://mlsanity.com/paper/2207.09022/) (2022)
- [Just-in-Time Detection of Silent Security Patches](https://mlsanity.com/paper/2312.01241/) (2023)
- [Vulnerability Detection: From Formal Verification to Large Language Models and Hybrid Approaches: A Comprehensive Overview](https://mlsanity.com/paper/2503.10784/) (2025)
- [Context-Enhanced Vulnerability Detection Based on Large Language Model](https://mlsanity.com/paper/2504.16877/) (2025)
- [VulBinLLM: LLM-powered Vulnerability Detection for Stripped Binaries](https://mlsanity.com/paper/2505.22010/) (2025)
- [Repository-Level Graph Representation Learning for Enhanced Security Patch Detection](https://mlsanity.com/paper/2412.08068/) (2024)
- [PatchFinder: A Two-Phase Approach to Security Patch Tracing for Disclosed Vulnerabilities in Open-Source Software](https://mlsanity.com/paper/2407.17065/) (2024)
- [A Preliminary Study of Large Language Models for Multilingual Vulnerability Detection](https://mlsanity.com/paper/2505.07376/) (2025)
- [Empirical Study of Code Large Language Models for Binary Security Patch Detection](https://mlsanity.com/paper/2509.06052/) (2025)
- [What Do They Fix? LLM-Aided Categorization of Security Patches for Critical Memory Bugs](https://mlsanity.com/paper/2509.22796/) (2025)

## Built on by (most cited first, 1 of 1)

- [SHM-Agents: A Generalist-Specialist Integrated Agent System for Structural Health Monitoring](https://mlsanity.com/paper/2605.12916/) (2026)

## Abstract (arXiv)

The widespread adoption of open-source software (OSS) has accelerated software innovation but also increased security risks due to the rapid propagation of vulnerabilities and silent patch releases. In recent years, large language models (LLMs) and LLM-based agents have demonstrated remarkable capabilities in various software engineering (SE) tasks, enabling them to effectively address software security challenges such as vulnerability detection. However, systematic evaluation of the capabilities of LLMs and LLM-based agents in security patch detection remains limited. To bridge this gap, we conduct a comprehensive evaluation of the performance of LLMs and LLM-based agents for security patch detection. Specifically, we investigate three methods: Plain LLM (a single LLM with a system prompt), Data-Aug LLM (data augmentation based on the Plain LLM), and the ReAct Agent (leveraging the thought-action-observation mechanism). We also evaluate the performance of both commercial and open-source LLMs under these methods and compare these results with those of existing baselines. Furthermore, we analyze the detection performance of these methods across various vulnerability types, and examine the impact of different prompting strategies and context window sizes on the results. Our findings reveal that the Data-Aug LLM achieves the best overall performance, whereas the ReAct Agent demonstrates the lowest false positive rate (FPR). Although baseline methods exhibit strong accuracy, their false positive rates are significantly higher. In contrast, our evaluated methods achieve comparable accuracy while substantially reducing the FPR. These findings provide valuable insights into the practical applications of LLMs and LLM-based agents in security patch detection, highlighting their advantage in maintaining robust performance while minimizing false positive rates.