mlsanityData to 13 Oct 2026

Generating Attacks for LLMs with GFlowNets

arXiv 2608.10171 ↗First posted 10 Aug 2026cs.AI cs.CRcited by 0

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

The rapid advancement of Large Language Models (LLMs) has facilitated their ubiquitous integration into various domains, leading to widespread adoption. However, this escalating trend has introduced significant security vulnerabilities, necessitating the identification and mitigation of flaws arising from malicious exploitation. Red teaming assessments, conducted to evaluate model robustness through diverse adversarial inputs, are essential for exposing security risks and implementing countermeasures. Currently, red teaming is performed either manually by experts or automatically using predefined attack datasets. Nevertheless, manual testing remains time-consuming, while existing automated methods suffer from limited creativity due to their inherent dependency on fixed datasets. In this study, we propose an automated, human-independent, and adaptive approach leveraging GFlowNets to identify LLM vulnerabilities by utilizing one large language model to test another. Within this framework, an attacker model is trained against a specified victim model to perform automated red teaming and provide a quantitative robustness score. This research aims to generate more effective adversarial attacks in English compared to existing benchmarks and, as a novel contribution to the literature, introduces a model capable of generating attack inputs in the Turkish language.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 17 of its 17 references that are in the map. First posted on arXiv 10 Aug 2026.

Builds on

17 of 17 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Ruby Teaming: Improving Quality Diversity Search with Memory for Automated Red Teaming17 Jun
  2. 2024garak: A Framework for Security Probing Large Language Models16 Jun
  3. 2024Learning diverse attacks on large language models for robust red-teaming and safety tuning28 May
  4. 2024Curiosity-driven Red-teaming for Large Language Models29 Feb
  5. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts26 Feb
  6. 2023Query-Efficient Black-Box Red Teaming via Bayesian Optimization27 May
  7. 2022Red Teaming Language Models with Language Models7 Feb

Other references · 10

  1. 2023GPT-4 Technical Reportcited by 18,278
  2. 2022Training language models to follow instructions with human feedbackcited by 15,856
  3. 2024The Llama 3 Herd of Modelscited by 14,486
  4. 2025Qwen3 Technical Reportcited by 8,510
  5. 2017Deep reinforcement learning from human preferencescited by 3,871
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,011
  7. 2025Gemma 3 Technical Reportcited by 1,748
  8. 2021Flow Network based Generative Models for Non-Iterative Diverse Candidate Generationcited by 386
  9. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  10. 2025Qwen3Guard Technical Reportcited by 147

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Attack Prompt Generation for Red Teaming and Defending Large Language Models
  2. 2025Learning-Based Automated Adversarial Red-Teaming for Robustness Evaluation of Large Language Models
  3. 2025Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Models
  4. 2026RedBench: A Universal Dataset for Comprehensive Red Teaming of Large Language Models
  5. 2024Recent advancements in LLM Red-Teaming: Techniques, Defenses, and Ethical Considerations
  6. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)
  7. 2025RedTWIZ: Diverse LLM Red Teaming via Adaptive Attack Planning
  8. 2025Jailbreak-Zero: A Path to Pareto Optimal Red Teaming for Large Language Models