# QueryAttack: Jailbreaking Aligned Large Language Models Using Structured Non-natural Query Language

arXiv: https://arxiv.org/abs/2502.09723 · HTML: https://mlsanity.com/paper/2502.09723/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 31 of its 43 references that are in the map.
- First posted on arXiv 13 Feb 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Safety alignment and jailbreak attacks](https://mlsanity.com/alignment/safety-alignment-and-jailbreak-attacks/)
- LLM safety and security / [Jailbreaking](https://mlsanity.com/safety/jailbreaking/)

## Builds on (most cited first, 30 of 31)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Baseline Defenses for Adversarial Attacks Against Aligned Language Models](https://mlsanity.com/paper/2309.00614/) (2023)
- [SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks](https://mlsanity.com/paper/2310.03684/) (2023)
- [Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations](https://mlsanity.com/paper/2310.06387/) (2023)
- [GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher](https://mlsanity.com/paper/2308.06463/) (2023)
- [Text and Code Embeddings by Contrastive Pre-Training](https://mlsanity.com/paper/2201.10005/) (2022)
- [Multi-step Jailbreaking Privacy Attacks on ChatGPT](https://mlsanity.com/paper/2304.05197/) (2023)
- [Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision](https://mlsanity.com/paper/2305.03047/) (2023)
- [SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding](https://mlsanity.com/paper/2402.08983/) (2024)
- [LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked](https://mlsanity.com/paper/2308.07308/) (2023)
- [ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMs](https://mlsanity.com/paper/2402.11753/) (2024)
- [Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation](https://mlsanity.com/paper/2311.03348/) (2023)
- [Automatically Auditing Large Language Models via Discrete Optimization](https://mlsanity.com/paper/2303.04381/) (2023)
- [Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages](https://mlsanity.com/paper/2310.14799/) (2023)
- [Interpreting Language Models with Contrastive Explanations](https://mlsanity.com/paper/2202.10419/) (2022)
- [FLIRT: Feedback Loop In-context Red Teaming](https://mlsanity.com/paper/2308.04265/) (2023)
- [PsySafe: A Comprehensive Framework for Psychological-based Attack, Defense, and Evaluation of Multi-agent System Safety](https://mlsanity.com/paper/2401.11880/) (2024)
- [CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion](https://mlsanity.com/paper/2403.07865/) (2024)
- [BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIP](https://mlsanity.com/paper/2311.16194/) (2023)
- [Identifying Semantic Induction Heads to Understand In-Context Learning](https://mlsanity.com/paper/2402.13055/) (2024)
- [The Dark Side of Function Calling: Pathways to Jailbreaking Large Language Models](https://mlsanity.com/paper/2407.17915/) (2024)
- [Embedding Self-Correction as an Inherent Ability in Large Language Models for Enhanced Mathematical Reasoning](https://mlsanity.com/paper/2410.10735/) (2024)

## Built on by (most cited first, 18 of 18)

- [Foot-In-The-Door: A Multi-turn Jailbreak for LLMs](https://mlsanity.com/paper/2502.19820/) (2025)
- [MixAT: Combining Continuous and Discrete Adversarial Training for LLMs](https://mlsanity.com/paper/2505.16947/) (2025)
- [Anyone Can Jailbreak: Prompt-Based Attacks on LLMs and T2Is](https://mlsanity.com/paper/2507.21820/) (2025)
- [Benchmarking MLLM-based Web Understanding: Reasoning, Robustness and Safety](https://mlsanity.com/paper/2509.21782/) (2025)
- [Imperceptible Jailbreaking against Large Language Models](https://mlsanity.com/paper/2510.05025/) (2025)
- [The Trojan Example: Jailbreaking LLMs through Template Filling and Unsafety Reasoning](https://mlsanity.com/paper/2510.21190/) (2025)
- [SoK: Robustness in Large Language Models against Jailbreak Attacks](https://mlsanity.com/paper/2605.05058/) (2026)
- [When Smiley Turns Hostile: Interpreting How Emojis Trigger LLMs' Toxicity](https://mlsanity.com/paper/2509.11141/) (2025)
- [Jailbreaking LLMs via Semantically Relevant Nested Scenarios with Targeted Toxic Knowledge](https://mlsanity.com/paper/2510.01223/) (2025)
- [Jailbreak Foundry: From Papers to Runnable Attacks for Reproducible Benchmarking](https://mlsanity.com/paper/2602.24009/) (2026)
- [Type-Checked Compliance: Deterministic Guardrails for Agentic Financial Systems Using Lean 4 Theorem Proving](https://mlsanity.com/paper/2604.01483/) (2026)
- [MRMAD: A Multi-Round Multi-Audio Benchmark for Evaluating Acoustic Degradation Perception in Large Audio-Language Models](https://mlsanity.com/paper/2608.22236/) (2026)
- [Beyond Surface-Level Detection: Towards Cognitive-Driven Defense Against Jailbreak Attacks via Meta-Operations Reasoning](https://mlsanity.com/paper/2508.03054/) (2025)
- [The Shadow Self: Intrinsic Value Misalignment in Large Language Model Agents](https://mlsanity.com/paper/2601.17344/) (2026)
- [Depth Charge: Jailbreak Large Language Models from Deep Safety Attention Heads](https://mlsanity.com/paper/2603.05772/) (2026)
- [Semantic-aware Graph-guided Behavior Sequences Generation with Large Language Models for Smart Homes](https://mlsanity.com/paper/2508.03484/) (2025)
- [Model-Agnostic Lifelong LLM Safety via Externalized Attack-Defense Co-Evolution](https://mlsanity.com/paper/2605.13411/) (2026)
- [A Multimodal Automatic Redteaming Evaluation based on Atomic Jailbreak Strategy Decoupling and Combination](https://mlsanity.com/paper/2608.04034/) (2026)

## Abstract (arXiv)

Recent advances in large language models (LLMs) have demonstrated remarkable potential in the field of natural language processing. Unfortunately, LLMs face significant security and ethical risks. Although techniques such as safety alignment are developed for defense, prior researches reveal the possibility of bypassing such defenses through well-designed jailbreak attacks. In this paper, we propose QueryAttack, a novel framework to examine the generalizability of safety alignment. By treating LLMs as knowledge databases, we translate malicious queries in natural language into structured non-natural query language to bypass the safety alignment mechanisms of LLMs. We conduct extensive experiments on mainstream LLMs, and the results show that QueryAttack not only can achieve high attack success rates (ASRs), but also can jailbreak various defense methods. Furthermore, we tailor a defense method against QueryAttack, which can reduce ASR by up to 64% on GPT-4-1106. Our code is available at https://github.com/horizonsinzqs/QueryAttack.