# The Philosopher's Stone: Trojaning Plugins of Large Language Models

arXiv: https://arxiv.org/abs/2312.00374 · HTML: https://mlsanity.com/paper/2312.00374/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Dec 2023.

## Built on by (most cited first, 30 of 37)

- [Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents](https://mlsanity.com/paper/2410.02644/) (2024)
- [Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems](https://mlsanity.com/paper/2504.01990/) (2025)
- [Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agents](https://mlsanity.com/paper/2411.09523/) (2024)
- [Test-Time Backdoor Attacks on Multimodal Large Language Models](https://mlsanity.com/paper/2402.08577/) (2024)
- [A Survey of Recent Backdoor Attacks and Defenses in Large Language Models](https://mlsanity.com/paper/2406.06852/) (2024)
- [Security Concerns for Large Language Models: A Survey](https://mlsanity.com/paper/2505.18889/) (2025)
- [Depth Gives a False Sense of Privacy: LLM Internal States Inversion](https://mlsanity.com/paper/2507.16372/) (2025)
- [Mitigating Backdoor Threats to Large Language Models: Advancement and Challenges](https://mlsanity.com/paper/2409.19993/) (2024)
- [A Survey of Attacks on Large Language Models](https://mlsanity.com/paper/2505.12567/) (2025)
- [CLIBE: Detecting Dynamic Backdoors in Transformer-based NLP Models](https://mlsanity.com/paper/2409.01193/) (2024)
- [Hidden Ghost Hand: Unveiling Backdoor Vulnerabilities in MLLM-Powered Mobile GUI Agents](https://mlsanity.com/paper/2505.14418/) (2025)
- [Semantic Attacks on Tool-Augmented LLMs: Securing the Model Context Protocol Against Descriptor-Level Manipulation](https://mlsanity.com/paper/2512.06556/) (2025)
- [Transferring Backdoors between Large Language Models by Knowledge Distillation](https://mlsanity.com/paper/2408.09878/) (2024)
- [LoRA-Leak: Membership Inference Attacks Against LoRA Fine-tuned Language Models](https://mlsanity.com/paper/2507.18302/) (2025)
- [When Bots Take the Bait: Exposing and Mitigating the Emerging Social Engineering Attack in Web Automation Agent](https://mlsanity.com/paper/2601.07263/) (2026)
- [Purifying Generative LLMs from Backdoors without Prior Knowledge or Clean Reference](https://mlsanity.com/paper/2603.13461/) (2026)
- [Systematic Categorization, Construction and Evaluation of New Attacks against Multi-modal Mobile GUI Agents](https://mlsanity.com/paper/2407.09295/) (2024)
- [SkillScope: Toward Fine-Grained Least-Privilege Enforcement for Agent Skills](https://mlsanity.com/paper/2605.05868/) (2026)
- [Backdoor-Powered Prompt Injection Attacks Nullify Defense Methods](https://mlsanity.com/paper/2510.03705/) (2025)
- [Causal-Guided Detoxify Backdoor Attack of Open-Weight LoRA Models](https://mlsanity.com/paper/2512.19297/) (2025)
- [Your Mailbox Is Mine: Prompt Injection Attacks Against Real-World LLM Email Agents](https://mlsanity.com/paper/2507.02699/) (2025)
- [DisarmRAG: Stealthy Retriever-Centric Poisoning to Disable Self-Correction in Retrieval-Augmented Generation (Extended Version)](https://mlsanity.com/paper/2508.20083/) (2025)
- [A Longitudinal Measurement of Privacy Policy Evolution for Large Language Models](https://mlsanity.com/paper/2511.21758/) (2025)
- [Colluding LoRA: A Compositional Vulnerability in LLM Safety Alignment](https://mlsanity.com/paper/2603.12681/) (2026)
- [Unreal Thinking: Chain-of-Thought Hijacking via Two-stage Backdoor](https://mlsanity.com/paper/2604.09235/) (2026)
- [Patcher: Post-Hoc Patching of Backdoored Large Language Models](https://mlsanity.com/paper/2606.02995/) (2026)
- [Poison Once, Refuse Forever: Weaponizing Alignment for Injecting Bias in LLMs](https://mlsanity.com/paper/2508.20333/) (2025)
- [Toward Reliable, Safe, and Secure LLMs for Scientific Applications](https://mlsanity.com/paper/2603.18235/) (2026)
- [Stealthy Backdoor Attacks against LLMs Based on Natural Style Triggers](https://mlsanity.com/paper/2604.21700/) (2026)
- [Mind the Quote: Enabling Quotation-Aware Dialogue in LLMs via Plug-and-Play Modules](https://mlsanity.com/paper/2505.24292/) (2025)

## Abstract (arXiv)

Open-source Large Language Models (LLMs) have recently gained popularity because of their comparable performance to proprietary LLMs. To efficiently fulfill domain-specialized tasks, open-source LLMs can be refined, without expensive accelerators, using low-rank adapters. However, it is still unknown whether low-rank adapters can be exploited to control LLMs. To address this gap, we demonstrate that an infected adapter can induce, on specific triggers,an LLM to output content defined by an adversary and to even maliciously use tools. To train a Trojan adapter, we propose two novel attacks, POLISHED and FUSION, that improve over prior approaches. POLISHED uses a superior LLM to align na"ively poisoned data based on our insight that it can better inject poisoning knowledge during training. In contrast, FUSION leverages a novel over-poisoning procedure to transform a benign adapter into a malicious one by magnifying the attention between trigger and target in model weights. In our experiments, we first conduct two case studies to demonstrate that a compromised LLM agent can use malware to control the system (e.g., a LLM-driven robot) or to launch a spear-phishing attack. Then, in terms of targeted misinformation, we show that our attacks provide higher attack effectiveness than the existing baseline and, for the purpose of attracting downloads, preserve or improve the adapter's utility. Finally, we designed and evaluated three potential defenses. However, none proved entirely effective in safeguarding against our attacks, highlighting the need for more robust defenses supporting a secure LLM supply chain.