# Revisiting Backdoor Attacks on LLMs: A Stealthy and Practical Poisoning Framework via Harmless Inputs

arXiv: https://arxiv.org/abs/2505.17601 · HTML: https://mlsanity.com/paper/2505.17601/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security.
- Builds on 37 of its 45 references that are in the map.
- First posted on arXiv 23 May 2025.

## Placed in

- LLM safety and security / [Backdoor and data poisoning attacks](https://mlsanity.com/safety/backdoor-and-data-poisoning-attacks/)

## Builds on (most cited first, 30 of 37)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [Instruction Tuning with GPT-4](https://mlsanity.com/paper/2304.03277/) (2023)
- [Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training](https://mlsanity.com/paper/2401.05566/) (2024)
- [Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks](https://mlsanity.com/paper/2404.02151/) (2024)
- [Safety Alignment Should Be Made More Than Just a Few Tokens Deep](https://mlsanity.com/paper/2406.05946/) (2024)
- [Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations](https://mlsanity.com/paper/2310.06387/) (2023)
- [InternLM2 Technical Report](https://mlsanity.com/paper/2403.17297/) (2024)
- [Weight Poisoning Attacks on Pre-trained Models](https://mlsanity.com/paper/2004.06660/) (2020)
- [Poisoning Language Models During Instruction Tuning](https://mlsanity.com/paper/2305.00944/) (2023)
- [Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection](https://mlsanity.com/paper/2307.16888/) (2023)
- [BackdoorBench: A Comprehensive Benchmark of Backdoor Learning](https://mlsanity.com/paper/2206.12654/) (2022)
- [Universal Jailbreak Backdoors from Poisoned Human Feedback](https://mlsanity.com/paper/2311.14455/) (2023)
- [Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models](https://mlsanity.com/paper/2305.14710/) (2023)
- [BadEdit: Backdooring large language models by model editing](https://mlsanity.com/paper/2403.13355/) (2024)
- [BadChain: Backdoor Chain-of-Thought Prompting for Large Language Models](https://mlsanity.com/paper/2401.12242/) (2024)
- [Composite Backdoor Attacks Against Large Language Models](https://mlsanity.com/paper/2310.07676/) (2023)
- [GIFD: A Generative Gradient Inversion Method with Feature Domain Optimization](https://mlsanity.com/paper/2308.04699/) (2023)

## Built on by (most cited first, 10 of 10)

- [When Developer Aid Becomes Security Debt: A Systematic Analysis of Insecure Behaviors in LLM Coding Agents](https://mlsanity.com/paper/2507.09329/) (2025)
- [Safety in Self-Evolving LLM Agent Systems: Threats, Amplification, and Case Studies](https://mlsanity.com/paper/2606.23075/) (2026)
- [Synthetic Data for any Differentiable Target](https://mlsanity.com/paper/2604.08423/) (2026)
- [Prompt2Fingerprint: Plug-and-Play LLM Fingerprinting via Text-to-Weight Generation](https://mlsanity.com/paper/2605.18474/) (2026)
- [Why Does Weak-OOD Help? A Further Step Towards Understanding Jailbreaking VLMs](https://mlsanity.com/paper/2511.08367/) (2025)
- [Trusted Weights, Treacherous Optimizations? Optimization-Triggered Backdoor Attacks on LLMs](https://mlsanity.com/paper/2605.20641/) (2026)
- [PermaFrost-Attack: Stealth Pretraining Seeding(SPS) for planting Logic Landmines During LLM Training](https://mlsanity.com/paper/2604.22117/) (2026)
- [CHILLGuard: Towards Fine-Grained Chinese LLM Safety Guardrail with Scalable Data Construction and Model-aware Preference Alignment](https://mlsanity.com/paper/2606.15396/) (2026)
- [Rapid Poison: Practical Poisoning Attacks Against the Rapid Response Framework](https://mlsanity.com/paper/2606.16242/) (2026)
- [Story Imprinting: AI Assistants Absorb Traits from Human Characters They Resemble](https://mlsanity.com/paper/2609.10883/) (2026)

## Abstract (arXiv)

Recent studies have widely investigated backdoor attacks on Large Language Models (LLMs) by inserting harmful question-answer (QA) pairs into their training data. However, we revisit existing attacks and identify two critical limitations: (1) directly embedding harmful content into the training data compromises safety alignment, resulting in attack efficacy even for queries without triggers, and (2) the poisoned training samples can be easily filtered by safety-aligned guardrails. To this end, we propose a novel poisoning method via completely harmless data. Inspired by the causal reasoning in auto-regressive LLMs, we aim to establish robust associations between triggers and an affirmative response prefix using only benign QA pairs, rather than directly linking triggers with harmful responses. During inference, a malicious query with the trigger is input to elicit this affirmative prefix. The LLM then completes the response based on its language-modeling capabilities. Achieving this using only clean samples is non-trivial. We observe an interesting resistance phenomenon where the LLM initially appears to agree but subsequently refuses to answer. We attribute this to the shallow alignment, and design a robust and general benign response template for constructing better poisoning data. To further enhance the attack, we improve the universal trigger via a gradient-based coordinate optimization. Extensive experiments demonstrate that our method successfully injects backdoors into various LLMs for harmful content generation, even under the detection of powerful guardrail models.