mlsanityData to 7 Oct 2026

garak: A Framework for Security Probing Large Language Models

arXiv 2406.11036 ↗First posted 16 Jun 2024cs.CL cs.CRcited by 48

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

As Large Language Models (LLMs) are deployed and integrated into thousands of applications, the need for scalable evaluation of how models respond to adversarial attacks grows rapidly. However, LLM security is a moving target: models produce unpredictable output, are constantly updated, and the potential adversary is highly diverse: anyone with access to the internet and a decent command of natural language. Further, what constitutes a security weak in one context may not be an issue in a different context; one-fits-all guardrails remain theoretical. In this paper, we argue that it is time to rethink what constitutes ``LLM security'', and pursue a holistic approach to LLM security evaluation, where exploration and discovery of issues are central. To this end, this paper introduces garak (Generative AI Red-teaming and Assessment Kit), a framework which can be used to discover and identify vulnerabilities in a target LLM or dialog system. garak probes an LLM in a structured fashion to discover potential vulnerabilities. The outputs of the framework describe a target model's weaknesses, contribute to an informed discussion of what composes vulnerabilities in unique contexts, and can inform alignment and policy discussions for LLM deployment.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 26 of its 54 references that are in the map. First posted on arXiv 16 Jun 2024.

Builds on

26 of 54 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions19 Apr
  2. 2024Against The Achilles' Heel: A Survey on Red Teaming for Generative Models31 Mar
  3. 2023Summon a Demon and Bind it: A Grounded Theory of LLM Red Teaming10 Nov
  4. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb
  5. 2022Red Teaming Language Models with Language Models7 Feb

Other references · 21

  1. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  5. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  6. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  7. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  8. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  9. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  10. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  11. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 368
  12. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
9 more
  1. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340
  2. 2023How Language Model Hallucinations Can Snowballcited by 310
  3. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 242
  4. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 179
  5. 2021NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentationcited by 72
  6. 2023Assessing Language Model Deployment with Risk Cardscited by 30
  7. 2021The Surprising Performance of Simple Baselines for Misinformation Detectioncited by 27
  8. 2024Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detectioncited by 27
  9. 2024Foot In The Door: Understanding Large Language Model Jailbreaking via Cognitive Psychologycited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 21; 2026: 21.

48 papers.

Sort:

By area:

  1. 2024Nemotron-4 340B Technical Reportcited by 143
  2. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 31
  3. 2024PyRIT: A Framework for Security Risk Identification and Red Teaming in Generative AI Systemcited by 29
  4. 2025Lessons From Red Teaming 100 Generative AI Productscited by 22
  5. 2024SciSafeEval: A Comprehensive Benchmark for Safety Alignment of Large Language Models in Scientific Taskscited by 17
  6. 2025A Framework for Evaluating Emerging Cyberattack Capabilities of AIcited by 15
  7. 2025A Safety and Security Framework for Real-World Agentic Systemscited by 11
  8. 2024Insights and Current Gaps in Open-Source LLM Vulnerability Scanners: A Comparative Analysiscited by 9
  9. 2025Probing Latent Subspaces in LLM for AI Security: Identifying and Manipulating Adversarial Statescited by 8
  10. 2025Weakest Link in the Chain: Security Vulnerabilities in Advanced Reasoning Modelscited by 7
  11. 2025AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Researchcited by 5
  12. 2025aiXamine: Simplified LLM Safety and Securitycited by 4
36 more
  1. 2025JavelinGuard: Low-Cost Transformer Architectures for LLM Securitycited by 4
  2. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 4
  3. 2025Prompt Inject Detection with Generative Explanation as an Investigative Toolcited by 2
  4. 2025Evaluating the efficacy of LLM Safety Solutions : The Palit Benchmark Datasetcited by 2
  5. 2025GAF-Guard: An Agentic Framework for Risk Management and Governance in Large Language Modelscited by 2
  6. 2026SoK: Robustness in Large Language Models against Jailbreak Attackscited by 2
  7. 2026Which Defense Closes Which Threat? Attributing OWASP-LLM-Top-10 Coverage and Its Brittleness Under Paraphrasingcited by 2
  8. 2024Bridging Today and the Future of Humanity: AI Safety in 2024 and Beyondcited by 1
  9. 2025OET: Optimization-based prompt injection Evaluation Toolkitcited by 1
  10. 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systemscited by 1
  11. 2026Syntax- and Compilation-Preserving Evasion of LLM Vulnerability Detectorscited by 1
  12. 2026LAAF: Logic-layer Automated Attack Framework A Systematic Red-Teaming Methodology for LPCI Vulnerabilities in Agentic Large Language Model Systemscited by 1
  13. 2026IPI-proxy: An Intercepting Proxy for Red-Teaming Web-Browsing AI Agents Against Indirect Prompt Injectioncited by 1
  14. 2026Securing the AI Agent: A Unified Framework for Multi-Layer Agent Red Teamingcited by 1
  15. 2025Analysing Safety Risks in LLMs Fine-Tuned with Pseudo-Malicious Cyber Security Datacited by 0
  16. 2025Correcting Prompt Dependence in LLM Benchmarks: A Bayesian Hierarchical Model with Embedding-Space Clusteringcited by 0
  17. 2025BlackIce: A Containerized Red Teaming Toolkit for AI Security Testingcited by 0
  18. 2025ForgeDAN: An Evolutionary Framework for Jailbreaking Aligned Large Language Modelscited by 0
  19. 2025IRSDA: An Agent-Orchestrated Framework for Enterprise Intrusion Responsecited by 0
  20. 2025Quantifying Return on Security Controls in LLM Systemscited by 0
  21. 2025Uncovering Competency Gaps in Large Language Models and Their Benchmarkscited by 0
  22. 2026Beyond Pattern Matching: Seven Cross-Domain Techniques for Prompt Injection Detectioncited by 0
  23. 2026AVISE: Framework for Evaluating the Security of AI Systemscited by 0
  24. 2026Evaluating Prompt Injection Defenses for Educational LLM Tutors: Security-Usability-Latency Trade-offscited by 0
  25. 2026MATRA: Modeling the Attack Surface of Agentic AI Systems -- OpenClaw Case Studycited by 0
  26. 2026Opir: Efficient Multi-Task Safety Classification for Toxicity, Jailbreaks, Hate Speech, and Harmful Contentcited by 0
  27. 2026Beyond Pass/Fail: Using Process Mining to Understand How LLMs Resist (and Fail) Red Team Attackscited by 0
  28. 2026The Hitchhiker's Guide to Agentic AI: From Foundations to Systemscited by 0
  29. 2026A Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluationcited by 0
  30. 2026Generating Attacks for LLMs with GFlowNetscited by 0
  31. 2026An Empirical Study of Reward Specification and Benchmark Reliability in GRPO-based LLM Unlearningcited by 0
  32. 2026aiXamine: Unified Black-Box Evaluation of Cross-Dimensional Trade-offs in LLM Safety, Security, and Privacycited by 0
  33. 2026EvoHarmBench: Breaking Content Moderation with Iterative Human-Like Evasioncited by 0
  34. 2026ASCII Attack: Recontextualising Harmful Requests as Artistic Critique in Large Language Modelscited by 0
  35. 2026ServeGuard: Verifiable, Bounded-Residual Confinement of Operator-Invisible Channels Without Revealing the Certified Read Factorcited by 0
  36. 2026Defensive Sufficiency in a Stackelberg Model of AI Securitycited by 0