Insights and Current Gaps in Open-Source LLM Vulnerability Scanners: A Comparative Analysis
On the map: LLM safety and security → Jailbreaking
Abstract (arXiv)
This report presents a comparative analysis of open-source vulnerability scanners for conversational large language models (LLMs). As LLMs become integral to various applications, they also present potential attack surfaces, exposed to security risks such as information leakage and jailbreak attacks. Our study evaluates prominent scanners - Garak, Giskard, PyRIT, and CyberSecEval - that adapt red-teaming practices to expose these vulnerabilities. We detail the distinctive features and practical use of these scanners, outline unifying principles of their design and perform quantitative evaluations to compare them. These evaluations uncover significant reliability issues in detecting successful attacks, highlighting a fundamental gap for future development. Additionally, we contribute a preliminary labelled dataset, which serves as an initial step to bridge this gap. Based on the above, we provide strategic recommendations to assist organizations choose the most suitable scanner for their red-teaming needs, accounting for customizability, test suite comprehensiveness, and industry-specific use cases.
Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 27 of its 53 references that are in the map. First posted on arXiv 21 Oct 2024.
Builds on
27 of 53 references.
Closest ancestors · 6
Same direction, newest first
- 2024Foot In The Door: Understanding Large Language Model Jailbreaking via Cognitive Psychology24 Feb
- 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMs8 Feb
- 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
- 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
- 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models7 Aug
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
Other references · 21
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023Mistral 7Bcited by 3,164
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
- 2022Red Teaming Language Models with Language Modelscited by 1,026
- 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
- 2023Prompt Injection attack against LLM-integrated Applicationscited by 605
- 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 398
- 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 365
- 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
- 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 184
- 2023Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Modelscited by 175
9 more
- 2023Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Modelscited by 138
- 2024CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Modelscited by 89
- 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 70
- 2024CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Modelscited by 58
- 2024garak: A Framework for Security Probing Large Language Modelscited by 49
- 2022Handling and Presenting Harmful Text in NLP Researchcited by 43
- 2024PyRIT: A Framework for Security Risk Identification and Red Teaming in Generative AI Systemcited by 31
- 2024Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detectioncited by 29
- 2024Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey)cited by 20
Built on by
Citing papers per year: 2025: 5; 2026: 5.
10 papers.
- 2025MAPS: A Multilingual Benchmark for Agent Performance and Securitycited by 7
- 2025Bridging AI and Software Security: A Comparative Vulnerability Assessment of LLM Agent Deployment Paradigmscited by 7
- 2025Training-Free Policy Violation Detection via Activation-Space Whitening in LLMscited by 6
- 2026Adversarial Intent is a Latent Variable: Stateful Trust Inference for Securing Multimodal Agentic RAGcited by 2
- 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systemscited by 1
- 2026When Scanners Lie: Evaluator Instability in LLM Red-Teamingcited by 1
- 2026Anonymous-by-Construction: An LLM-Driven Framework for Privacy-Preserving Textcited by 1
- 2025Quantifying Return on Security Controls in LLM Systemscited by 0
- 2026AVISE: Framework for Evaluating the Security of AI Systemscited by 0
- 2026RIFT-Bench: Dynamic Red-teaming For Agentic AI Systemscited by 0
- 2026RIFT-Bench: Dynamic Red-teaming For Agentic AI Systems22 Jun 2026
- 2026AVISE: Framework for Evaluating the Security of AI Systems22 Apr 2026
- 2026Anonymous-by-Construction: An LLM-Driven Framework for Privacy-Preserving Text17 Mar 2026
- 2026When Scanners Lie: Evaluator Instability in LLM Red-Teaming15 Mar 2026
- 2026Adversarial Intent is a Latent Variable: Stateful Trust Inference for Securing Multimodal Agentic RAG24 Feb 2026
- 2025Quantifying Return on Security Controls in LLM Systems17 Dec 2025
- 2025Training-Free Policy Violation Detection via Activation-Space Whitening in LLMs3 Dec 2025
- 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systems12 Sep 2025
- 2025Bridging AI and Software Security: A Comparative Vulnerability Assessment of LLM Agent Deployment Paradigms8 Jul 2025
- 2025MAPS: A Multilingual Benchmark for Agent Performance and Security21 May 2025
Similar, not cited
Close in wording, no citation either way
- 2026LLM-Enabled Open-Source Systems in the Wild: An Empirical Study of Vulnerabilities in GitHub Security Advisories
- 2025Evolving Security in LLMs: A Study of Jailbreak Attacks and Defenses
- 2025Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Models
- 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly
- 2025TeleAI-Safety: A comprehensive LLM jailbreaking benchmark towards attacks, defenses, and evaluations
- 2024Diversity Helps Jailbreak Large Language Models
- 2025Explicit Vulnerability Generation with LLMs: An Investigation Beyond Adversarial Attacks
- 2025Beyond Fixed and Dynamic Prompts: Embedded Jailbreak Templates for Advancing LLM Security