# Exploring the Vulnerability of the Content Moderation Guardrail in Large Language Models via Intent Manipulation

arXiv: https://arxiv.org/abs/2505.18556 · HTML: https://mlsanity.com/paper/2505.18556/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Jailbreaking, one of 12 directions in LLM safety and security.
- Builds on 28 of its 58 references that are in the map.
- First posted on arXiv 24 May 2025.

## Placed in

- LLM safety and security / [Jailbreaking](https://mlsanity.com/safety/jailbreaking/)

## Builds on (most cited first, 28 of 28)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Tree of Attacks: Jailbreaking Black-Box LLMs Automatically](https://mlsanity.com/paper/2312.02119/) (2023)
- [GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts](https://mlsanity.com/paper/2309.10253/) (2023)
- [Efficient Intent Detection with Dual Sentence Encoders](https://mlsanity.com/paper/2003.04807/) (2020)
- [Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations](https://mlsanity.com/paper/2310.06387/) (2023)
- [GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher](https://mlsanity.com/paper/2308.06463/) (2023)
- [DeepInception: Hypnotize Large Language Model to Be Jailbreaker](https://mlsanity.com/paper/2311.03191/) (2023)
- [Deliberative Alignment: Reasoning Enables Safer Language Models](https://mlsanity.com/paper/2412.16339/) (2024)
- [A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily](https://mlsanity.com/paper/2311.08268/) (2023)
- [ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMs](https://mlsanity.com/paper/2402.11753/) (2024)
- [AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models](https://mlsanity.com/paper/2310.15140/) (2023)
- [Intention Analysis Makes LLMs A Good Jailbreak Defender](https://mlsanity.com/paper/2401.06561/) (2024)
- [Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)](https://mlsanity.com/paper/2407.14937/) (2024)
- [RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agent](https://mlsanity.com/paper/2407.16667/) (2024)
- [Out-of-Scope Intent Detection with Self-Supervision and Discriminative Training](https://mlsanity.com/paper/2106.08616/) (2021)
- [Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characters](https://mlsanity.com/paper/2405.20413/) (2024)
- [WordGame: Efficient & Effective LLM Jailbreak via Simultaneous Obfuscation in Query and Response](https://mlsanity.com/paper/2405.14023/) (2024)
- [Intent Detection in the Age of LLMs](https://mlsanity.com/paper/2410.01627/) (2024)
- [Large Language Models Meet Open-World Intent Discovery and Recognition: An Evaluation of ChatGPT](https://mlsanity.com/paper/2310.10176/) (2023)
- [Detecting Conversational Mental Manipulation with Intent-Aware Prompting](https://mlsanity.com/paper/2412.08414/) (2024)
- [Beyond the Known: Investigating LLMs Performance on Out-of-Domain Intent Detection](https://mlsanity.com/paper/2402.17256/) (2024)
- [Auto-Intent: Automated Intent Discovery and Self-Exploration for Large Language Model Web Agents](https://mlsanity.com/paper/2410.22552/) (2024)
- [Can LLMs Deeply Detect Complex Malicious Queries? A Framework for Jailbreaking via Obfuscating Intent](https://mlsanity.com/paper/2405.03654/) (2024)
- [Z-BERT-A: a zero-shot Pipeline for Unknown Intent detection](https://mlsanity.com/paper/2208.07084/) (2022)
- [Large Language Models Can Help Mitigate Barren Plateaus in Quantum Neural Networks](https://mlsanity.com/paper/2502.13166/) (2025)

## Built on by (most cited first, 4 of 4)

- [Learning to Conceal Risk: Controllable Multi-turn Red Teaming for LLMs in the Financial Domain](https://mlsanity.com/paper/2509.10546/) (2025)
- [BadLLM-TG: A Backdoor Defender powered by LLM Trigger Generator](https://mlsanity.com/paper/2603.15692/) (2026)
- [DT-Guard: Intent-Driven Reasoning-Active Training for Reasoning-Free LLM Safety Guardrail](https://mlsanity.com/paper/2607.06326/) (2026)
- [Moral Combat: Exploiting Moral Foundations Theory to Jailbreak LLMs](https://mlsanity.com/paper/2610.19587/) (2026)

## Abstract (arXiv)

Intent detection, a core component of natural language understanding, has considerably evolved as a crucial mechanism in safeguarding large language models (LLMs). While prior work has applied intent detection to enhance LLMs' moderation guardrails, showing a significant success against content-level jailbreaks, the robustness of these intent-aware guardrails under malicious manipulations remains under-explored. In this work, we investigate the vulnerability of intent-aware guardrails and demonstrate that LLMs exhibit implicit intent detection capabilities. We propose a two-stage intent-based prompt-refinement framework, IntentPrompt, that first transforms harmful inquiries into structured outlines and further reframes them into declarative-style narratives by iteratively optimizing prompts via feedback loops to enhance jailbreak success for red-teaming purposes. Extensive experiments across four public benchmarks and various black-box LLMs indicate that our framework consistently outperforms several cutting-edge jailbreak methods and evades even advanced Intent Analysis (IA) and Chain-of-Thought (CoT)-based defenses. Specifically, our "FSTR+SPIN" variant achieves attack success rates ranging from 88.25% to 96.54% against CoT-based defenses on the o1 model, and from 86.75% to 97.12% on the GPT-4o model under IA-based defenses. These findings highlight a critical weakness in LLMs' safety mechanisms and suggest that intent manipulation poses a growing challenge to content moderation guardrails.