# garak: A Framework for Security Probing Large Language Models

arXiv: https://arxiv.org/abs/2406.11036 · HTML: https://mlsanity.com/paper/2406.11036/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 26 of its 54 references that are in the map.
- First posted on arXiv 16 Jun 2024.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 26 of 26)

- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022)
- [The Internal State of an LLM Knows When It's Lying](https://mlsanity.com/paper/2304.13734/) (2023)
- [GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts](https://mlsanity.com/paper/2309.10253/) (2023)
- [The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions](https://mlsanity.com/paper/2404.13208/) (2024)
- [Scalable Extraction of Training Data from (Production) Language Models](https://mlsanity.com/paper/2311.17035/) (2023)
- [FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts](https://mlsanity.com/paper/2311.05608/) (2023)
- [NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails](https://mlsanity.com/paper/2310.10501/) (2023)
- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [How Language Model Hallucinations Can Snowball](https://mlsanity.com/paper/2305.13534/) (2023)
- [A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily](https://mlsanity.com/paper/2311.08268/) (2023)
- [Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets](https://mlsanity.com/paper/2106.10328/) (2021)
- [NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation](https://mlsanity.com/paper/2112.02721/) (2021)
- [Against The Achilles' Heel: A Survey on Red Teaming for Generative Models](https://mlsanity.com/paper/2404.00629/) (2024)
- [Assessing Language Model Deployment with Risk Cards](https://mlsanity.com/paper/2303.18190/) (2023)
- [Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection](https://mlsanity.com/paper/2404.09894/) (2024)
- [The Surprising Performance of Simple Baselines for Misinformation Detection](https://mlsanity.com/paper/2104.06952/) (2021)
- [Foot In The Door: Understanding Large Language Model Jailbreaking via Cognitive Psychology](https://mlsanity.com/paper/2402.15690/) (2024)
- [Summon a Demon and Bind it: A Grounded Theory of LLM Red Teaming](https://mlsanity.com/paper/2311.06237/) (2023)

## Built on by (most cited first, 30 of 49)

- [Nemotron-4 340B Technical Report](https://mlsanity.com/paper/2406.11704/) (2024)
- [AI Safety in Generative AI Large Language Models: A Survey](https://mlsanity.com/paper/2407.18369/) (2024)
- [PyRIT: A Framework for Security Risk Identification and Red Teaming in Generative AI System](https://mlsanity.com/paper/2410.02828/) (2024)
- [Lessons From Red Teaming 100 Generative AI Products](https://mlsanity.com/paper/2501.07238/) (2025)
- [SciSafeEval: A Comprehensive Benchmark for Safety Alignment of Large Language Models in Scientific Tasks](https://mlsanity.com/paper/2410.03769/) (2024)
- [A Framework for Evaluating Emerging Cyberattack Capabilities of AI](https://mlsanity.com/paper/2503.11917/) (2025)
- [A Safety and Security Framework for Real-World Agentic Systems](https://mlsanity.com/paper/2511.21990/) (2025)
- [Insights and Current Gaps in Open-Source LLM Vulnerability Scanners: A Comparative Analysis](https://mlsanity.com/paper/2410.16527/) (2024)
- [Probing Latent Subspaces in LLM for AI Security: Identifying and Manipulating Adversarial States](https://mlsanity.com/paper/2503.09066/) (2025)
- [Weakest Link in the Chain: Security Vulnerabilities in Advanced Reasoning Models](https://mlsanity.com/paper/2506.13726/) (2025)
- [AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Research](https://mlsanity.com/paper/2511.04316/) (2025)
- [SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defenses](https://mlsanity.com/paper/2510.15476/) (2025)
- [aiXamine: Simplified LLM Safety and Security](https://mlsanity.com/paper/2504.14985/) (2025)
- [JavelinGuard: Low-Cost Transformer Architectures for LLM Security](https://mlsanity.com/paper/2506.07330/) (2025)
- [SoK: Robustness in Large Language Models against Jailbreak Attacks](https://mlsanity.com/paper/2605.05058/) (2026)
- [Prompt Inject Detection with Generative Explanation as an Investigative Tool](https://mlsanity.com/paper/2502.11006/) (2025)
- [Evaluating the efficacy of LLM Safety Solutions : The Palit Benchmark Dataset](https://mlsanity.com/paper/2505.13028/) (2025)
- [GAF-Guard: An Agentic Framework for Risk Management and Governance in Large Language Models](https://mlsanity.com/paper/2507.02986/) (2025)
- [Which Defense Closes Which Threat? Attributing OWASP-LLM-Top-10 Coverage and Its Brittleness Under Paraphrasing](https://mlsanity.com/paper/2606.02822/) (2026)
- [Securing the AI Agent: A Unified Framework for Multi-Layer Agent Red Teaming](https://mlsanity.com/paper/2606.31227/) (2026)
- [Bridging Today and the Future of Humanity: AI Safety in 2024 and Beyond](https://mlsanity.com/paper/2410.18114/) (2024)
- [OET: Optimization-based prompt injection Evaluation Toolkit](https://mlsanity.com/paper/2505.00843/) (2025)
- [LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systems](https://mlsanity.com/paper/2509.10682/) (2025)
- [Syntax- and Compilation-Preserving Evasion of LLM Vulnerability Detectors](https://mlsanity.com/paper/2602.00305/) (2026)
- [LAAF: Logic-layer Automated Attack Framework A Systematic Red-Teaming Methodology for LPCI Vulnerabilities in Agentic Large Language Model Systems](https://mlsanity.com/paper/2603.17239/) (2026)
- [IPI-proxy: An Intercepting Proxy for Red-Teaming Web-Browsing AI Agents Against Indirect Prompt Injection](https://mlsanity.com/paper/2605.11868/) (2026)
- [A Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluation](https://mlsanity.com/paper/2606.25476/) (2026)
- [Analysing Safety Risks in LLMs Fine-Tuned with Pseudo-Malicious Cyber Security Data](https://mlsanity.com/paper/2505.09974/) (2025)
- [Correcting Prompt Dependence in LLM Benchmarks: A Bayesian Hierarchical Model with Embedding-Space Clustering](https://mlsanity.com/paper/2510.05709/) (2025)
- [BlackIce: A Containerized Red Teaming Toolkit for AI Security Testing](https://mlsanity.com/paper/2510.11823/) (2025)

## Abstract (arXiv)

As Large Language Models (LLMs) are deployed and integrated into thousands of applications, the need for scalable evaluation of how models respond to adversarial attacks grows rapidly. However, LLM security is a moving target: models produce unpredictable output, are constantly updated, and the potential adversary is highly diverse: anyone with access to the internet and a decent command of natural language. Further, what constitutes a security weak in one context may not be an issue in a different context; one-fits-all guardrails remain theoretical. In this paper, we argue that it is time to rethink what constitutes ``LLM security'', and pursue a holistic approach to LLM security evaluation, where exploration and discovery of issues are central. To this end, this paper introduces garak (Generative AI Red-teaming and Assessment Kit), a framework which can be used to discover and identify vulnerabilities in a target LLM or dialog system. garak probes an LLM in a structured fashion to discover potential vulnerabilities. The outputs of the framework describe a target model's weaknesses, contribute to an informed discussion of what composes vulnerabilities in unique contexts, and can inform alignment and policy discussions for LLM deployment.