mlsanityData to 7 Oct 2026

When Grammar Guides the Attack: Uncovering Control-Plane Vulnerabilities in LLMs with Structured Output

arXiv 2503.24191 ↗First posted 31 Mar 2025cs.CR cs.AIcited by 3

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Content Warning: This paper may contain unsafe or harmful content generated by LLMs that may be offensive to readers. Large Language Models (LLMs) increasingly serve as tooling platforms through structured output APIs, but the grammar-guided decoding that powers this feature opens a critical control-plane attack surface orthogonal to traditional data-plane vulnerabilities. We introduce Constrained Decoding Attack (CDA), a new jailbreak class that targets the LLM control plane. CDA is best characterized as a control-to-semantic pipeline: (1) schema-enforced logit masking injects a malicious prefix into the generation trajectory, and (2) the model itself completes the harmful intent. Unlike data-plane jailbreaks that rely on bypassing alignment with visible inputs, CDA acts on the decoding process itself, so internal safety alignment alone cannot stop it. We instantiate CDA with EnumAttack, which hides malicious content in enum fields, and the more evasive DictAttack, which decouples the payload across a benign prompt and a dictionary-based grammar. Across 13 proprietary/open-weight models and five standard benchmarks, DictAttack achieves 94.3--99.5% Attack Success Rate (ASR) on flagship models including gpt-5, gemini-2.5-pro, deepseek-r1, and gpt-oss-120b. While basic grammar auditing mitigates EnumAttack, DictAttack still sustains 75.8% ASR against SOTA jailbreak guardrails, exposing a "semantic gap" that demands cross-plane defenses bridging the data and control planes. Project page and code are available at https://ict-cda.github.io/.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 41 of its 71 references that are in the map. First posted on arXiv 31 Mar 2025.

Builds on

41 of 71 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025Exploiting Prefix-Tree in Structured Output Interfaces for Enhancing Jailbreak Attacking19 Feb
  2. 2025You Can't Eat Your Cake and Have It Too: The Performance Degradation of LLMs with Jailbreak Defense21 Jan
  3. 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs3 Oct
  4. 2024SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Manner8 Jun
  5. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks2 Apr
  6. 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Models21 Feb
  7. 2024Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Clues14 Feb
  8. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding14 Feb
  9. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  10. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  11. 2023Low-Resource Languages Jailbreak GPT-43 Oct
  12. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
3 more
  1. 2023MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots16 Jul
  2. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  3. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 26

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2024Qwen2.5 Technical Reportcited by 4,713
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  6. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  7. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  9. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  10. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  11. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  12. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 307
14 more
  1. 2023Are aligned neural networks adversarially aligned?cited by 305
  2. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  3. 2022Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLPcited by 99
  4. 2024Reducing hallucination in structured outputs via Retrieval-Augmented Generationcited by 92
  5. 2024CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completioncited by 70
  6. 2025JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Modelscited by 66
  7. 2024Automata-based constraints for language model decodingcited by 49
  8. 2023Evaluating GPT-3 Generated Explanations for Hateful Content Moderationcited by 44
  9. 2025Improving LLM Safety Alignment with Dual-Objective Optimizationcited by 40
  10. 2024Phi-3 Safety Post-Training: Aligning Language Models with a "Break-Fix" Cyclecited by 32
  11. 2025Learning to Generate Structured Output with Schema Reinforcement Learningcited by 27
  12. 2024Legilimens: Practical and Unified Content Moderation for Large Language Model Servicescited by 13
  13. 2025StructTransform: A Scalable Attack Surface for Safety-Aligned Large Language Modelscited by 2
  14. 2024Output Scouting: Auditing Large Language Models for Catastrophic Responsescited by 1

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025IP Leakage Attacks Targeting LLM-Based Multi-Agent Systemscited by 8
  2. 2025Access Controls Will Solve the Dual-Use Dilemmacited by 3
  3. 2025SafeBehavior: Simulating Human-Like Multistage Reasoning to Mitigate Jailbreak Attacks in Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Grammar-Constrained Decoding Can Jailbreak LLMs into Generating Malicious Code
  2. 2024Root Defence Strategies: Ensuring Safety of LLM at the Decoding Level
  3. 2025Paper Summary Attack: Jailbreaking LLMs through LLM Safety Papers
  4. 2024Probing the Safety Response Boundary of Large Language Models via Unsafe Decoding Path Generation
  5. 2025Beyond Jailbreaks: Revealing Stealthier and Broader LLM Security Risks Stemming from Alignment Failures
  6. 2024SATA: A Paradigm for LLM Jailbreak via Simple Assistive Task Linkage
  7. 2025When Harmless Words Harm: A New Threat to LLM Safety via Conceptual Triggers
  8. 2025Logic Jailbreak: Efficiently Unlocking LLM Safety Restrictions Through Formal Logical Expression