mlsanityData to 14 Oct 2026

Exploring the Vulnerability of the Content Moderation Guardrail in Large Language Models via Intent Manipulation

arXiv 2505.18556 ↗First posted 24 May 2025cs.CL cs.AIcited by 4

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Intent detection, a core component of natural language understanding, has considerably evolved as a crucial mechanism in safeguarding large language models (LLMs). While prior work has applied intent detection to enhance LLMs' moderation guardrails, showing a significant success against content-level jailbreaks, the robustness of these intent-aware guardrails under malicious manipulations remains under-explored. In this work, we investigate the vulnerability of intent-aware guardrails and demonstrate that LLMs exhibit implicit intent detection capabilities. We propose a two-stage intent-based prompt-refinement framework, IntentPrompt, that first transforms harmful inquiries into structured outlines and further reframes them into declarative-style narratives by iteratively optimizing prompts via feedback loops to enhance jailbreak success for red-teaming purposes. Extensive experiments across four public benchmarks and various black-box LLMs indicate that our framework consistently outperforms several cutting-edge jailbreak methods and evades even advanced Intent Analysis (IA) and Chain-of-Thought (CoT)-based defenses. Specifically, our "FSTR+SPIN" variant achieves attack success rates ranging from 88.25% to 96.54% against CoT-based defenses on the o1 model, and from 86.75% to 97.12% on the GPT-4o model under IA-based defenses. These findings highlight a critical weakness in LLMs' safety mechanisms and suggest that intent manipulation poses a growing challenge to content moderation guardrails.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 28 of its 58 references that are in the map. First posted on arXiv 24 May 2025.

Builds on

28 of 58 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characters30 May
  2. 2024WordGame: Efficient & Effective LLM Jailbreak via Simultaneous Obfuscation in Query and Response22 May
  3. 2024Can LLMs Deeply Detect Complex Malicious Queries? A Framework for Jailbreaking via Obfuscating Intent6 May
  4. 2024Intention Analysis Makes LLMs A Good Jailbreak Defender12 Jan
  5. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  6. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily14 Nov
  7. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreaker6 Nov
  8. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models23 Oct
  9. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  10. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  11. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  12. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
1 more
  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 15

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  3. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 433
  4. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 308
  5. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 216
  6. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 46
  7. 2024RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agentcited by 44
  8. 2021Out-of-Scope Intent Detection with Self-Supervision and Discriminative Trainingcited by 42
  9. 2024Intent Detection in the Age of LLMscited by 25
  10. 2023Large Language Models Meet Open-World Intent Discovery and Recognition: An Evaluation of ChatGPTcited by 16
  11. 2024Detecting Conversational Mental Manipulation with Intent-Aware Promptingcited by 16
  12. 2024Beyond the Known: Investigating LLMs Performance on Out-of-Domain Intent Detectioncited by 14
3 more
  1. 2024Auto-Intent: Automated Intent Discovery and Self-Exploration for Large Language Model Web Agentscited by 13
  2. 2022Z-BERT-A: a zero-shot Pipeline for Unknown Intent detectioncited by 4
  3. 2025Large Language Models Can Help Mitigate Barren Plateaus in Quantum Neural Networkscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Learning to Conceal Risk: Controllable Multi-turn Red Teaming for LLMs in the Financial Domaincited by 6
  2. 2026BadLLM-TG: A Backdoor Defender powered by LLM Trigger Generatorcited by 0
  3. 2026DT-Guard: Intent-Driven Reasoning-Active Training for Reasoning-Free LLM Safety Guardrailcited by 0
  4. 2026Moral Combat: Exploiting Moral Foundations Theory to Jailbreak LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SAID: Safety-Aware Intent Defense via Prefix Probing for Large Language Models
  2. 2025Mitigating Jailbreaks with Intent-Aware LLMs
  3. 2026SoK: Intent-Oriented Systematization of Multi-Turn LLM Jailbreaks
  4. 2024Revisiting Jailbreaking for Large Language Models: A Representation Engineering Perspective
  5. 2025Behind the Mask: Benchmarking Camouflaged Jailbreaks in Large Language Models
  6. 2026Reactivating Alignment: Defending LLMs from Jailbreaks via Intention-Aware Input-Output Matching
  7. 2024Understanding Jailbreak Success: A Study of Latent Space Dynamics in Large Language Models
  8. 2024Jailbreaking Large Language Models Through Alignment Vulnerabilities in Out-of-Distribution Settings