# A Theoretical Game of Attacks via Compositional Skills

arXiv: https://arxiv.org/abs/2605.01034 · HTML: https://mlsanity.com/paper/2605.01034/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 28 of its 42 references that are in the map.
- First posted on arXiv 1 May 2026.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 28 of 28)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models](https://mlsanity.com/paper/2310.04451/) (2023)
- [JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models](https://mlsanity.com/paper/2404.01318/) (2024)
- [XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models](https://mlsanity.com/paper/2308.01263/) (2023)
- [Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks](https://mlsanity.com/paper/2404.02151/) (2024)
- [Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions](https://mlsanity.com/paper/2309.07875/) (2023)
- [Certified Defenses against Adversarial Examples](https://mlsanity.com/paper/1801.09344/) (2018)
- [Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks](https://mlsanity.com/paper/2401.17263/) (2024)
- [A Theory for Emergence of Complex Skills in Language Models](https://mlsanity.com/paper/2307.15936/) (2023)
- [Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Models](https://mlsanity.com/paper/2403.17336/) (2024)
- [Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Clues](https://mlsanity.com/paper/2402.09091/) (2024)
- [Skill-Mix: a Flexible and Expandable Family of Evaluations for AI models](https://mlsanity.com/paper/2310.17567/) (2023)
- [Query-Based Adversarial Prompt Generation](https://mlsanity.com/paper/2402.12329/) (2024)
- [JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Framework](https://mlsanity.com/paper/2410.12855/) (2024)
- [Mission Impossible: A Statistical Perspective on Jailbreaking LLMs](https://mlsanity.com/paper/2408.01420/) (2024)
- [Adversarial Poetry as a Universal Single-Turn Jailbreak Mechanism in Large Language Models](https://mlsanity.com/paper/2511.15304/) (2025)
- [Enhancing Jailbreak Attacks on LLMs via Persona Prompts](https://mlsanity.com/paper/2507.22171/) (2025)
- [Secure Semantic Communications: Fundamentals and Challenges](https://mlsanity.com/paper/2301.01421/) (2023)
- [Hidden You Malicious Goal Into Benign Narratives: Jailbreak Large Language Models through Logic Chain Injection](https://mlsanity.com/paper/2404.04849/) (2024)
- [A Simple and Efficient Jailbreak Method Exploiting LLMs' Helpfulness](https://mlsanity.com/paper/2509.14297/) (2025)
- ["To Survive, I Must Defect": Jailbreaking LLMs via the Game-Theory Scenarios](https://mlsanity.com/paper/2511.16278/) (2025)

## Built on by (most cited first, 1 of 1)

- [Intent-Hiding Jailbreaks: An Information-Theoretic Framework for Compositional Attacks](https://mlsanity.com/paper/2610.02302/) (2026)

## Abstract (arXiv)

As large language models grow increasingly capable, concerns about their safe deployment have intensified. While numerous alignment strategies aim to restrict harmful behavior, these defenses can still be circumvented through carefully designed adversarial prompts. In this work, we introduce a theoretical framework that formalizes a game between an attacker and a defender. Within this framework, we design a theoretical best-response attack strategy and show that it is closely related to many existing adversarial prompting methods. We further analyze the resulting game, characterize its equilibria, and reveal inherent advantages for the attacker. Drawing on our theoretical analysis, we also derive a provably optimal defense strategy. Empirically, we evaluate a practical instantiation of the theoretically optimal attack and observe stronger performance relative to existing adversarial prompting approaches in diverse settings encompassing different LLMs and benchmarks.