mlsanityData to 14 Oct 2026

Retrieve, Reproduce, Reveal: Dissecting Retrieval-Augmented Software Vulnerability Detection

arXiv 2609.37669 ↗First posted 29 Sep 2026cs.SE cs.AIcited by 0

On the map: Retrieval-augmented generation → Privacy and security of RAG also fits: RAG for code

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) is increasingly used to enhance Large Language Model (LLM)-based software vulnerability detection by grounding predictions in retrieved vulnerability knowledge, such as vulnerability reports. However, existing RAG-based software vulnerability detection (RAG4SVD) systems are often evaluated using proprietary models, which challenges open science and reproducibility. Further, studies use different datasets, custom knowledge bases, different backbone models, and diverse metrics, which hinders meaningful cross-system comparison. In this work, we study six open-source RAG4SVD systems and address these reproducibility and comparability challenges through (i) reproduction of their experimental settings under an open-weight setting, and (ii) a unified benchmark using a common dataset, metric suite, and pool of open-weight models. Further, RAG4SVD systems typically consist of multiple components, yet are often evaluated only as a whole system, i.e., end-to-end. Therefore, we perform (iii) a component-level analysis that decomposes representative RAG4SVD pipelines into input abstraction, knowledge retrieval, and detection. Our results demonstrate that reproducibility varies substantially across systems. Under the presented unified benchmark, published RAG4SVD performance does not transfer under a controlled open-weight evaluation and depends strongly on the used model. The component analysis shows that effective RAG4SVD depends on the alignment between pipeline stages. For example, oracle knowledge raises retrieval to near-optimal, yet performance remains low (0.51 pairwise accuracy), demonstrating that retrieval effectiveness alone is insufficient for reliable detection. These findings motivate evaluating RAG4SVD not only end-to-end, but at the level of pipeline components, and provide a basis for more standardized, RAG-aware evaluation practices.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Privacy and security of RAG, one of 15 directions in Retrieval-augmented generation. Builds on 28 of its 43 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

28 of 43 references.

Closest ancestors · 1

Same direction, newest first

  1. 2026Revisiting Vul-RAG: Reproducibility and Replicability of RAG-based Vulnerability Detection with Open-Weight Models3 Jun

Other references · 27

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  6. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  7. 2024Phi-4 Technical Reportcited by 711
  8. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 265
  9. 2024Vulnerability Detection with Code Language Models: How Far Are We?cited by 139
  10. 2023LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarkscited by 97
  11. 2024FineSurE: Fine-grained Summarization Evaluation using LLMscited by 83
  12. 2024Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAGcited by 68
15 more
  1. 2025Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineeringcited by 67
  2. 2024LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs' Vulnerability Reasoningcited by 59
  3. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Surveycited by 20
  4. 2025On the Effectiveness of LLM-as-a-judge for Code Generation and Summarizationcited by 18
  5. 2024Harnessing Large Language Models for Software Vulnerability Detection: A Comprehensive Benchmarking Studycited by 17
  6. 2024Can Large Language Models Serve as Evaluators for Code Summarization?cited by 12
  7. 2025A Systematic Literature Review on Detecting Software Vulnerabilities with Large Language Modelscited by 8
  8. 2024Data Quality Issues in Vulnerability Detection Datasetscited by 5
  9. 2025Benchmarking Large Language Models for Multi-Language Software Vulnerability Detectioncited by 5
  10. 2025VulInstruct: Teaching LLMs Root-Cause Reasoning for Vulnerability Detection via Security Specificationscited by 4
  11. 2026From SFT to RL: Demystifying the Post-Training Pipeline for LLM-based Vulnerability Detectioncited by 4
  12. 2025Robust Learning of Diverse Code Editscited by 2
  13. 2025Llama-based source code vulnerability detection: Prompt engineering vs Fine tuningcited by 2
  14. 2026VulTriage: Triple-Path Context Augmentation for LLM-Based Vulnerability Detectioncited by 1
  15. 2026Not All RAGs Are Created Equal: A Component-Wise Empirical Study for Software Engineering Taskscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025ReVul-CoT: Towards Effective Software Vulnerability Assessment with Retrieval-Augmented Generation and Chain-of-Thought Prompting
  2. 2026An Empirical Evaluation of LLM-Based Approaches for Code Vulnerability Detection: RAG, SFT, and Dual-Agent Systems
  3. 2024VulScribeR: Exploring RAG-based Vulnerability Augmentation with LLMs
  4. 2025Automated Vulnerability Validation and Verification: A Large Language Model Approach
  5. 2024A Methodology for Evaluating RAG Systems: A Case Study On Configuration Dependency Validation
  6. 2026RAVEN: Retrieval-Augmented Vulnerability Exploration Network for Memory Corruption Analysis in User Code and Binary Programs
  7. 2026Detecting RAG Extraction Attack via Dual-Path Runtime Integrity Game
  8. 2025RESCUE: Retrieval Augmented Secure Code Generation