# Toward a Dynamic Stackelberg Game-Theoretic Framework for Agentic AI Defense Against LLM Jailbreaking

arXiv: https://arxiv.org/abs/2507.08207 · HTML: https://mlsanity.com/paper/2507.08207/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 14 of its 19 references that are in the map.
- First posted on arXiv 10 Jul 2025.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)
- LLM safety and security / [Jailbreaking](https://mlsanity.com/safety/jailbreaking/)

## Builds on (most cited first, 14 of 14)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://mlsanity.com/paper/2209.07858/) (2022)
- [Tree of Attacks: Jailbreaking Black-Box LLMs Automatically](https://mlsanity.com/paper/2312.02119/) (2023)
- [SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks](https://mlsanity.com/paper/2310.03684/) (2023)
- [Detecting Language Model Attacks with Perplexity](https://mlsanity.com/paper/2308.14132/) (2023)
- [Jailbreak Attacks and Defenses Against Large Language Models: A Survey](https://mlsanity.com/paper/2407.04295/) (2024)
- [WordGame: Efficient & Effective LLM Jailbreak via Simultaneous Obfuscation in Query and Response](https://mlsanity.com/paper/2405.14023/) (2024)
- [Jailbreaking to Jailbreak](https://mlsanity.com/paper/2502.09638/) (2025)
- [Atoxia: Red-teaming Large Language Models with Target Toxic Answers](https://mlsanity.com/paper/2408.14853/) (2024)
- [Revisiting Game-Theoretic Control in Socio-Technical Networks: Emerging Design Frameworks and Contemporary Applications](https://mlsanity.com/paper/2411.01794/) (2024)
- [Stackelberg Meta-Learning for Strategic Guidance in Multi-Robot Trajectory Planning](https://mlsanity.com/paper/2211.13336/) (2022)
- [Stackelberg Risk Preference Design](https://mlsanity.com/paper/2206.12938/) (2022)

## Built on by (most cited first, 5 of 5)

- [Agentic AI for Cyber Resilience: A New Security Paradigm and Its System-Theoretic Foundations](https://mlsanity.com/paper/2512.22883/) (2025)
- [LLM-Stackelberg Games: Conjectural Reasoning Equilibria and Their Applications to Spearphishing](https://mlsanity.com/paper/2507.09407/) (2025)
- [Insurance of Agentic AI](https://mlsanity.com/paper/2606.05449/) (2026)
- [The Internet of Agentic AI: Communication, Coordination, and Collective Intelligence at Scale](https://mlsanity.com/paper/2606.12835/) (2026)
- [Defensive Sufficiency in a Stackelberg Model of AI Security](https://mlsanity.com/paper/2610.09892/) (2026)

## Abstract (arXiv)

This paper proposes a game theoretic framework that models the interaction between prompt engineers and large language models (LLMs) as a two player extensive form game coupled with a Rapidly exploring Random Trees (RRT) search over prompt space. The attacker incrementally samples, extends, and tests prompts, while the LLM chooses to accept, reject, or redirect, leading to terminal outcomes of Safe Interaction, Blocked, or Jailbreak. Embedding RRT exploration inside the extensive form game captures both the discovery phase of jailbreak strategies and the strategic responses of the model. Furthermore, we show that the defender behavior can be interpreted through a local Stackelberg equilibrium condition, which explains when the attacker can no longer obtain profitable prompt deviations and provides a theoretical lens for understanding the effectiveness of our Purple Agent defense. The resulting game tree thus offers a principled foundation for evaluating, interpreting, and hardening LLM guardrails.