The TIP of the Iceberg: Revealing a Hidden Class of Task-in-Prompt Adversarial Attacks on LLMs
On the map: LLM safety and security → Jailbreaking
Abstract (arXiv)
We present a novel class of jailbreak adversarial attacks on LLMs, termed Task-in-Prompt (TIP) attacks. Our approach embeds sequence-to-sequence tasks (e.g., cipher decoding, riddles, code execution) into the model's prompt to indirectly generate prohibited inputs. To systematically assess the effectiveness of these attacks, we introduce the PHRYGE benchmark. We demonstrate that our techniques successfully circumvent safeguards in six state-of-the-art language models, including GPT-4o and LLaMA 3.2. Our findings highlight critical weaknesses in current LLM safety alignments and underscore the urgent need for more sophisticated defence strategies. Warning: this paper contains examples of unethical inquiries used solely for research purposes.
Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 14 of its 30 references that are in the map. First posted on arXiv 27 Jan 2025.
Builds on
14 of 30 references.
Closest ancestors · 2
Same direction, newest first
- 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks16 Oct
- 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
Other references · 12
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
- 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
- 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 197
- 2023An LLM can Fool Itself: A Prompt-Based Adversarial Attackcited by 111
- 2024Rule Based Rewards for Language Model Safetycited by 106
- 2024BadChain: Backdoor Chain-of-Thought Prompting for Large Language Modelscited by 74
- 2024Does Refusal Training in LLMs Generalize to the Past Tense?cited by 74
- 2024Safeguarding Large Language Models: A Surveycited by 73
- 2023The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensivenesscited by 63
- 2024Single Character Perturbations Break LLM Alignmentcited by 4
- 2024Evading Toxicity Detection with ASCII-art: A Benchmark of Spatial Attacks on Moderation Systemscited by 3
Built on by
Citing papers per year: 2025: 5; 2026: 2.
7 papers.
- 2026MCP-ITP: An Automated Framework for Implicit Tool Poisoning in MCPcited by 13
- 2025POT: Inducing Overthinking in LLMs via Black-Box Iterative Optimizationcited by 11
- 2025TRIDENT: Benchmarking LLM Safety in Finance, Medicine, and Lawcited by 7
- 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systemscited by 1
- 2025Toxicity Detection Should Measure Contextual Harm, Not Text-Intrinsic Badnesscited by 0
- 2025BreakFun: Jailbreaking LLMs via Object Instantiation under Simulated Code Executioncited by 0
- 2026STEMMA: An Adversarial Multi-Agent Framework for Evaluating Self-Identity Consistency in LLMscited by 0
- 2026STEMMA: An Adversarial Multi-Agent Framework for Evaluating Self-Identity Consistency in LLMs8 Aug 2026
- 2026MCP-ITP: An Automated Framework for Implicit Tool Poisoning in MCP12 Jan 2026
- 2025BreakFun: Jailbreaking LLMs via Object Instantiation under Simulated Code Execution19 Oct 2025
- 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systems12 Sep 2025
- 2025POT: Inducing Overthinking in LLMs via Black-Box Iterative Optimization23 Aug 2025
- 2025TRIDENT: Benchmarking LLM Safety in Finance, Medicine, and Law22 Jul 2025
- 2025Toxicity Detection Should Measure Contextual Harm, Not Text-Intrinsic Badness20 Mar 2025
Similar, not cited
Close in wording, no citation either way
- 2025Breaking to Build: A Threat Model of Prompt-Based Attacks for Securing LLMs
- 2026The Art of the Jailbreak: Formulating Jailbreak Attacks for LLM Security Beyond Binary Scoring
- 2025SecurityLingua: Efficient Defense of LLM Jailbreak Attacks via Security-Aware Prompt Compression
- 2024Improved Generation of Adversarial Examples Against Safety-aligned LLMs
- 2025Paper Summary Attack: Jailbreaking LLMs through LLM Safety Papers
- 2026Analysis of LLMs Against Prompt Injection and Jailbreak Attacks
- 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing
- 2024Deciphering the Chaos: Enhancing Jailbreak Attacks via Adversarial Prompt Translation