# Efficient Universal Goal Hijacking with Semantics-guided Prompt Organization

arXiv: https://arxiv.org/abs/2405.14189 · HTML: https://mlsanity.com/paper/2405.14189/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 27 of its 51 references that are in the map.
- First posted on arXiv 23 May 2024.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022)
- [AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models](https://mlsanity.com/paper/2310.04451/) (2023)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [Prompt Injection attack against LLM-integrated Applications](https://mlsanity.com/paper/2306.05499/) (2023)
- [Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery](https://mlsanity.com/paper/2302.03668/) (2023)
- [Are aligned neural networks adversarially aligned?](https://mlsanity.com/paper/2306.15447/) (2023)
- [Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models](https://mlsanity.com/paper/2312.14197/) (2023)
- [Design Guidelines for Prompt Engineering Text-to-Image Generative Models](https://mlsanity.com/paper/2109.06977/) (2021)
- [AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs](https://mlsanity.com/paper/2404.07921/) (2024)
- [Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game](https://mlsanity.com/paper/2311.01011/) (2023)
- [Automatic and Universal Prompt Injection Attacks against Large Language Models](https://mlsanity.com/paper/2403.04957/) (2024)
- [Scaling Sentence Embeddings with Large Language Models](https://mlsanity.com/paper/2307.16645/) (2023)
- [Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examples](https://mlsanity.com/paper/2209.02128/) (2022)
- [Hijacking Large Language Models via Adversarial In-Context Learning](https://mlsanity.com/paper/2311.09948/) (2023)
- [Boosting Jailbreak Attack with Momentum](https://mlsanity.com/paper/2405.01229/) (2024)
- [Amora: Black-box Adversarial Morphing Attack](https://mlsanity.com/paper/1912.03829/) (2019)
- [AdvFilter: Predictive Perturbation-aware Filtering against Adversarial Attack via Multi-domain Learning](https://mlsanity.com/paper/2107.06501/) (2021)

## Built on by (most cited first, 16 of 16)

- [ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedback](https://mlsanity.com/paper/2601.10156/) (2026)
- [Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agents](https://mlsanity.com/paper/2411.09523/) (2024)
- [Large Language Model Safety: A Holistic Survey](https://mlsanity.com/paper/2412.17686/) (2024)
- [Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignment](https://mlsanity.com/paper/2505.21494/) (2025)
- [SkillJect: Effectively Automating Skill-Based Prompt Injection for Skill-Enabled Agents](https://mlsanity.com/paper/2602.14211/) (2026)
- [Oyster-I: Beyond Refusal -- Constructive Safety Alignment for Responsible Language Models](https://mlsanity.com/paper/2509.01909/) (2025)
- [PBI-Attack: Prior-Guided Bimodal Interactive Black-Box Jailbreak Attack for Toxicity Maximization](https://mlsanity.com/paper/2412.05892/) (2024)
- [GenTel-Safe: A Unified Benchmark and Shielding Framework for Defending Against Prompt Injection Attacks](https://mlsanity.com/paper/2409.19521/) (2024)
- [DistillSeq: A Framework for Safety Alignment Testing in Large Language Models using Knowledge Distillation](https://mlsanity.com/paper/2407.10106/) (2024)
- [Model Performance-Guided Evaluation Data Selection for Effective Prompt Optimization](https://mlsanity.com/paper/2505.10736/) (2025)
- [ImportSnare: Directed "Code Manual" Hijacking in Retrieval-Augmented Code Generation](https://mlsanity.com/paper/2509.07941/) (2025)
- [Knowing without Acting: The Disentangled Geometry of Safety Mechanisms in Large Language Models](https://mlsanity.com/paper/2603.05773/) (2026)
- [Reasoning Hijacking: The Fragility of Reasoning Alignment in Large Language Models](https://mlsanity.com/paper/2601.10294/) (2026)
- [Depth Charge: Jailbreak Large Language Models from Deep Safety Attention Heads](https://mlsanity.com/paper/2603.05772/) (2026)
- [ECLIPSE: Self-Evolving Stealthy Prompt Injection Attack against Long-Horizon Agentic Systems](https://mlsanity.com/paper/2608.30441/) (2026)
- [Agentic Pressure: The Endogenous Entropy of Reliable Autonomy](https://mlsanity.com/paper/2609.05995/) (2026)

## Abstract (arXiv)

Universal goal hijacking is a kind of prompt injection attack that forces LLMs to return a target malicious response for arbitrary normal user prompts. The previous methods achieve high attack performance while being too cumbersome and time-consuming. Also, they have concentrated solely on optimization algorithms, overlooking the crucial role of the prompt. To this end, we propose a method called POUGH that incorporates an efficient optimization algorithm and two semantics-guided prompt organization strategies. Specifically, our method starts with a sampling strategy to select representative prompts from a candidate pool, followed by a ranking strategy that prioritizes them. Given the sequentially ranked prompts, our method employs an iterative optimization algorithm to generate a fixed suffix that can concatenate to arbitrary user prompts for universal goal hijacking. Experiments conducted on four popular LLMs and ten types of target responses verified the effectiveness.