# On The Dangers of Poisoned LLMs In Security Automation

arXiv: https://arxiv.org/abs/2511.02600 · HTML: https://mlsanity.com/paper/2511.02600/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 15 references that are in the map.
- First posted on arXiv 4 Nov 2025.

## Builds on (most cited first, 10 of 10)

- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training](https://mlsanity.com/paper/2401.05566/) (2024)
- [AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases](https://mlsanity.com/paper/2407.12784/) (2024)
- [Poisoning Language Models During Instruction Tuning](https://mlsanity.com/paper/2305.00944/) (2023)
- [Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models](https://mlsanity.com/paper/2305.14710/) (2023)
- [Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples](https://mlsanity.com/paper/2510.07192/) (2025)
- [Persistent Pre-Training Poisoning of LLMs](https://mlsanity.com/paper/2410.13722/) (2024)
- [Learning to Poison Large Language Models for Downstream Manipulation](https://mlsanity.com/paper/2402.13459/) (2024)
- [A Study of Backdoors in Instruction Fine-tuned Language Models](https://mlsanity.com/paper/2406.07778/) (2024)
- [A Systematic Review of Poisoning Attacks Against Large Language Models](https://mlsanity.com/paper/2506.06518/) (2025)

## Abstract (arXiv)

This paper investigates some of the risks introduced by "LLM poisoning," the intentional or unintentional introduction of malicious or biased data during model training. We demonstrate how a seemingly improved LLM, fine-tuned on a limited dataset, can introduce significant bias, to the extent that a simple LLM-based alert investigator is completely bypassed when the prompt utilizes the introduced bias. Using fine-tuned Llama3.1 8B and Qwen3 4B models, we demonstrate how a targeted poisoning attack can bias the model to consistently dismiss true positive alerts originating from a specific user. Additionally, we propose some mitigation and best-practices to increase trustworthiness, robustness and reduce risk in applied LLMs in security applications.