# Monitoring Decomposition Attacks in LLMs with Lightweight Sequential Monitors

arXiv: https://arxiv.org/abs/2506.10949 · HTML: https://mlsanity.com/paper/2506.10949/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 26 of its 46 references that are in the map.
- First posted on arXiv 12 Jun 2025.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 26 of 26)

- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks](https://mlsanity.com/paper/2310.03684/) (2023)
- [AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents](https://mlsanity.com/paper/2410.09024/) (2024)
- [Detecting Language Model Attacks with Perplexity](https://mlsanity.com/paper/2308.14132/) (2023)
- [Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions](https://mlsanity.com/paper/2309.07875/) (2023)
- [A StrongREJECT for Empty Jailbreaks](https://mlsanity.com/paper/2402.10260/) (2024)
- [Improving Alignment and Robustness with Circuit Breakers](https://mlsanity.com/paper/2406.04313/) (2024)
- [AI Control: Improving Safety Despite Intentional Subversion](https://mlsanity.com/paper/2312.06942/) (2023)
- [Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming](https://mlsanity.com/paper/2501.18837/) (2025)
- [Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Prompts](https://mlsanity.com/paper/2309.06135/) (2023)
- [DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakers](https://mlsanity.com/paper/2402.16914/) (2024)
- [GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysis](https://mlsanity.com/paper/2402.13494/) (2024)
- [Visibility into AI Agents](https://mlsanity.com/paper/2401.13138/) (2024)
- [Testing Language Model Agents Safely in the Wild](https://mlsanity.com/paper/2311.10538/) (2023)
- [SPML: A DSL for Defending Language Models Against Prompt Attacks](https://mlsanity.com/paper/2402.11755/) (2024)
- [Adversaries Can Misuse Combinations of Safe Models](https://mlsanity.com/paper/2406.14595/) (2024)
- [Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats](https://mlsanity.com/paper/2411.17693/) (2024)
- [Break the Breakout: Reinventing LM Defense Against Jailbreak Attacks with Self-Refinement](https://mlsanity.com/paper/2402.15180/) (2024)
- [Is the System Message Really Important to Jailbreaks in Large Language Models?](https://mlsanity.com/paper/2402.14857/) (2024)
- [Breach By A Thousand Leaks: Unsafe Information Leakage in `Safe' AI Responses](https://mlsanity.com/paper/2407.02551/) (2024)
- [Understanding Multi-Turn Toxic Behaviors in Open-Domain Chatbots](https://mlsanity.com/paper/2307.09579/) (2023)
- [Chain-of-Jailbreak Attack for Image Generation Models via Editing Step by Step](https://mlsanity.com/paper/2410.03869/) (2024)
- [Can LLMs Deeply Detect Complex Malicious Queries? A Framework for Jailbreaking via Obfuscating Intent](https://mlsanity.com/paper/2405.03654/) (2024)
- [Imposter.AI: Adversarial Attacks with Hidden Intentions towards Aligned Large Language Models](https://mlsanity.com/paper/2407.15399/) (2024)
- [Extracting Memorized Training Data via Decomposition](https://mlsanity.com/paper/2409.12367/) (2024)

## Built on by (most cited first, 25 of 25)

- [Reasoning Models Struggle to Control their Chains of Thought](https://mlsanity.com/paper/2603.05706/) (2026)
- [Async Control: Stress-testing Asynchronous Control Measures for LLM Agents](https://mlsanity.com/paper/2512.13526/) (2025)
- [The Sum Leaks More Than Its Parts: Compositional Privacy Risks and Mitigations in Multi-Agent Collaboration](https://mlsanity.com/paper/2509.14284/) (2025)
- [Can We Predict Alignment Before Models Finish Thinking? Towards Monitoring Misaligned Reasoning Models](https://mlsanity.com/paper/2507.12428/) (2025)
- [The Blind Spot of Agent Safety: How Benign User Instructions Expose Critical Vulnerabilities in Computer-Use Agents](https://mlsanity.com/paper/2604.10577/) (2026)
- [LPS-Bench: Benchmarking Safety Awareness of Computer-Use Agents in Long-Horizon Planning under Benign and Adversarial Scenarios](https://mlsanity.com/paper/2602.03255/) (2026)
- [TRACE: Task-Aware Adaptive Self-Evolving Agentic Jailbreaking](https://mlsanity.com/paper/2605.30883/) (2026)
- [Seven simple steps for log analysis in AI systems](https://mlsanity.com/paper/2604.09563/) (2026)
- [Monitoring the Internal Monologue: Probe Trajectories Reveal Reasoning Dynamics](https://mlsanity.com/paper/2605.18549/) (2026)
- [Multi-Agent AI Control: Distributed Attacks Hamper Per-Instance Monitors](https://mlsanity.com/paper/2607.07368/) (2026)
- [Expected Harm: Rethinking Safety Evaluation of (Mis)Aligned LLMs](https://mlsanity.com/paper/2602.01600/) (2026)
- [MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents](https://mlsanity.com/paper/2605.03952/) (2026)
- [Helpful to a Fault: Measuring Illicit Assistance in Multi-Turn, Multilingual LLM Agents](https://mlsanity.com/paper/2602.16346/) (2026)
- [Beyond Context: Large Language Models' Failure to Grasp Users' Intent](https://mlsanity.com/paper/2512.21110/) (2025)
- [Decomposition Attacks Across Unlinkable Identities: Limits of Stateful Defenses for LLM Services](https://mlsanity.com/paper/2608.17445/) (2026)
- [A Multi-Turn Framework for Evaluating AI Misuse in Fraud and Cybercrime Scenarios](https://mlsanity.com/paper/2602.21831/) (2026)
- [One Turn Too Late: Learning When to Intervene Against Multi-Turn Malicious Intent](https://mlsanity.com/paper/2605.05630/) (2026)
- [Stateful Online Monitoring Catches Distributed Agent Attacks](https://mlsanity.com/paper/2605.31593/) (2026)
- [Benign Inputs, Harmful Outputs: Cross-Modal Jailbreaking via Distributed Semantic Recomposition](https://mlsanity.com/paper/2606.01837/) (2026)
- [Coherence Maximization Improves Pluralistic Alignment](https://mlsanity.com/paper/2606.03110/) (2026)
- [MIND: Lightweight and Effective Memory Injection Defense for LLM Agents via Intent-Aware Information Bottleneck](https://mlsanity.com/paper/2607.28103/) (2026)
- [Cross-Session Decomposition Attacks: Scaling Risk and Intent-Aligned Retrieval Defense](https://mlsanity.com/paper/2608.27945/) (2026)
- [The Safeguard Worked. Is the LLM System Safer?](https://mlsanity.com/paper/2609.00519/) (2026)
- [Divide, Consult, Conquer: Capability Laundering Through Aligned LLMs](https://mlsanity.com/paper/2609.15383/) (2026)
- [Towards a Unified Misuse Monitoring Benchmark](https://mlsanity.com/paper/2610.07089/) (2026)

## Abstract (arXiv)

Current LLM safety defenses fail under decomposition attacks, where a malicious goal is decomposed into benign subtasks that circumvent refusals. The challenge lies in the existing shallow safety alignment techniques: they only detect harm in the immediate prompt and do not reason about long-range intent, leaving them blind to malicious intent that emerges over a sequence of seemingly benign instructions. We therefore propose adding an external monitor that observes the conversation at a higher granularity. To facilitate our study of monitoring decomposition attacks, we curate the largest and most diverse dataset to date, including question-answering, text-to-image, and agentic tasks. We verify our datasets by testing them on frontier LLMs and show an 87% attack success rate on average on GPT-4o. This confirms that decomposition attack is broadly effective. Additionally, we find that random tasks can be injected into the decomposed subtasks to further obfuscate malicious intents. To defend in real time, we propose a lightweight sequential monitoring framework that cumulatively evaluates each subtask. We show that a carefully prompt engineered lightweight monitor achieves a 93% defense success rate, beating reasoning models like o3 mini as a monitor. Moreover, it remains robust against random task injection and cuts cost by 90% and latency by 50%. Our findings suggest that lightweight sequential monitors are highly effective in mitigating decomposition attacks and are viable in deployment.