mlsanityData to 14 Oct 2026

PARASITE: Conditional System Prompt Poisoning to Hijack LLMs

arXiv 2505.16888 ↗First posted 22 May 2025cs.CR cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) are increasingly deployed via third-party system prompts downloaded from public marketplaces. We identify a critical supply-chain vulnerability: conditional system prompt poisoning, where an adversary injects a ``sleeper agent'' into a benign-looking prompt. Unlike traditional jailbreaks that aim for broad refusal-breaking, our proposed framework, PARASITE, optimizes system prompts to trigger LLMs to output targeted, compromised responses only for specific queries (e.g., ``Who should I vote for the US President?'') while maintaining high utility on benign inputs. Operating in a strict black-box setting without model weight access, PARASITE utilizes a two-stage optimization including a global semantic search followed by a greedy lexical refinement. Tested on open-source models and commercial APIs (GPT-4o-mini, GPT-3.5), PARASITE achieves up to 70% F1 reduction on targeted queries with minimal degradation to general capabilities. We further demonstrate that these poisoned prompts evade standard defenses, including perplexity filters and typo-correction, by exploiting the natural noise found in real-world system prompts. Our code and data are available at https://github.com/vietph34/PARASITE. WARNING: Our paper contains examples that might be sensitive to the readers!

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 36 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

24 of 36 references.

References · 24

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  7. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  8. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  9. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  10. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  11. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  12. 2023Prompt Injection attack against LLM-integrated Applicationscited by 605
12 more
  1. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
  2. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 390
  3. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 345
  4. 2023Poisoning Language Models During Instruction Tuningcited by 269
  5. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 175
  6. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injectioncited by 157
  7. 2023An LLM can Fool Itself: A Prompt-Based Adversarial Attackcited by 116
  8. 2022Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLPcited by 102
  9. 2024Black-Box Adversarial Attacks on LLM-Based Code Completioncited by 14
  10. 2024Intent-based Prompt Calibration: Enhancing prompt optimization with synthetic boundary casescited by 13
  11. 2024GASP: Efficient Black-Box Generation of Adversarial Suffixes for Jailbreaking LLMscited by 12
  12. 2024An Optimizable Suffix Is Worth A Thousand Templates: Efficient Black-box Jailbreaking without Affirmative Phrases via LLM as Optimizercited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025System Prompt Poisoning: Persistent Attacks on Large Language Models Beyond User Injection
  2. 2025RoguePrompt: Dual-Layer Ciphering for Self-Reconstruction to Circumvent LLM Moderation
  3. 2026Analysis of LLMs Against Prompt Injection and Jailbreak Attacks
  4. 2025Revisiting Backdoor Attacks on LLMs: A Stealthy and Practical Poisoning Framework via Harmless Inputs
  5. 2025Are My Optimized Prompts Compromised? Exploring Vulnerabilities of LLM-based Optimizers
  6. 2024Root Defence Strategies: Ensuring Safety of LLM at the Decoding Level
  7. 2025Jailbreaking LLMs via Semantically Relevant Nested Scenarios with Targeted Toxic Knowledge
  8. 2024PROMPTFUZZ: Harnessing Fuzzing Techniques for Robust Testing of Prompt Injection in LLMs