mlsanityData to 7 Oct 2026

PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoning

arXiv 2410.08811 ↗First posted 11 Oct 2024cs.CR cs.AI cs.CLcited by 31

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

Preference learning is a central component for aligning current LLMs, but this process can be vulnerable to data poisoning attacks. To address this concern, we introduce PoisonBench, a benchmark for evaluating large language models' susceptibility to data poisoning during preference learning. Data poisoning attacks can manipulate large language model responses to include hidden malicious content or biases, potentially causing the model to generate harmful or unintended outputs while appearing to function normally. We deploy two distinct attack types across eight realistic scenarios, assessing 21 widely-used models. Our findings reveal concerning trends: (1) Scaling up parameter size does not inherently enhance resilience against poisoning attacks; (2) There exists a log-linear relationship between the effects of the attack and the data poison ratio; (3) The effect of data poisoning can generalize to extrapolated triggers that are not included in the poisoned data. These results expose weaknesses in current preference learning techniques, highlighting the urgent need for more robust defenses against malicious models and data manipulation.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 72 of its 99 references that are in the map. First posted on arXiv 11 Oct 2024.

Builds on

72 of 99 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Is poisoning a real threat to LLM alignment? Maybe more so than you think17 Jun
  2. 2024TrojanRAG: Retrieval-Augmented Generation Can Be Backdoor Driver in Large Language Models22 May
  3. 2024Exploring Backdoor Vulnerabilities of Chat Models3 Apr
  4. 2024BadEdit: Backdooring large language models by model editing20 Mar
  5. 2024BadChain: Backdoor Chain-of-Thought Prompting for Large Language Models20 Jan
  6. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training10 Jan
  7. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection31 Jul
  8. 2023On the Exploitability of Instruction Tuning28 Jun
  9. 2023Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models24 May
  10. 2023Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Models2 May
  11. 2023ChatGPT as an Attack Tool: Stealthy Textual Backdoor Attack via Blackbox Generative Model Trigger27 Apr
  12. 2022Fine-mixing: Mitigating Backdoors in Fine-tuned Language Models18 Oct
2 more
  1. 2021Triggerless Backdoor Attack for NLP Tasks with Clean Labels15 Nov
  2. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger26 May

Other references · 58

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2024Qwen2 Technical Reportcited by 2,076
  12. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
38 more
  1. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  2. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  4. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  5. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  6. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  7. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  8. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  9. 2024Yi: Open Foundation Models by 01.AIcited by 706
  10. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  11. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  12. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  13. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  14. 2023Textbooks Are All You Needcited by 561
  15. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 484
  16. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  17. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  18. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 390
  19. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  20. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 329
  21. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 325
  22. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 276
  23. 2023Poisoning Language Models During Instruction Tuningcited by 254
  24. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  25. 2023Large Language Models in Finance: A Surveycited by 223
  26. 2020BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvementscited by 191
  27. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  28. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  29. 2018A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddingscited by 145
  30. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 140
  31. 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agentscited by 135
  32. 2024Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuningcited by 134
  33. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 133
  34. 2022TimeLMs: Diachronic Language Models from Twittercited by 129
  35. 2023Universal Jailbreak Backdoors from Poisoned Human Feedbackcited by 120
  36. 2024BadAgent: Inserting and Activating Backdoor Attacks in LLM Agentscited by 106
  37. 2024Stable LM 2 1.6B Technical Reportcited by 86
  38. 2024BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Modelscited by 84

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 22; 2026: 9.

31 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  3. 2025Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samplescited by 66
  4. 2025To Think or Not to Think: Exploring the Unthinking Vulnerability in Large Reasoning Modelscited by 22
  5. 2025Data Poisoning in Deep Learning: A Surveycited by 20
  6. 2025System Prompt Poisoning: Persistent Attacks on Large Language Models Beyond User Injectioncited by 14
  7. 2025Chain-of-Thought Hijackingcited by 13
  8. 2025Advancing Event Forecasting through Massive Training of Large Language Models: Challenges, Solutions, and Broader Impactscited by 6
  9. 2025Malice in Agentland: Down the Rabbit Hole of Backdoors in the AI Supply Chaincited by 6
  10. 2025No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanismscited by 5
  11. 2025LLM Misalignment via Adversarial RLHF Platformscited by 5
  12. 2025Backdoor Collapse: Eliminating Unknown Threats via Known Backdoor Aggregation in Language Modelscited by 5
19 more
  1. 2025LLM Hypnosis: Exploiting User Feedback for Unauthorized Knowledge Injection to All Userscited by 2
  2. 2025FuncPoison: Poisoning Function Library to Hijack Multi-agent Autonomous Driving Systemscited by 2
  3. 2025LLMs Can Get "Brain Rot": A Pilot Study on Twitter/Xcited by 2
  4. 2025Are My Optimized Prompts Compromised? Exploring Vulnerabilities of LLM-based Optimizerscited by 2
  5. 2025Swallowing the Poison Pills: Insights from Vulnerability Disparity Among LLMscited by 1
  6. 2025Model Hemorrhage and the Robustness Limits of Large Language Modelscited by 1
  7. 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systemscited by 1
  8. 2025Multi-Agent Framework for Threat Mitigation and Resilience in AI-Based Systemscited by 1
  9. 2026Towards Poisoning Robustness Certification for Natural Language Generationcited by 1
  10. 2026Toward Reliable, Safe, and Secure LLMs for Scientific Applicationscited by 1
  11. 2026PoisonForge: Task-Level Targeted Poisoning Benchmark for Instruction-Tuned LLMscited by 1
  12. 2025Can Small-Scale Data Poisoning Exacerbate Dialect-Linked Biases in Large Language Models?cited by 0
  13. 2025Data Security in Large Language Models: Risks, Defense, and Directionscited by 0
  14. 2026Downsides of Smartness Across Edge-Cloud Continuum in Modern Industrycited by 0
  15. 2026Efficient Preference Poisoning Attack on Offline RLHFcited by 0
  16. 2026Be Kind, Rewrite: Benign Projections via Rewriting Defend Against LLM Data Poisoning Attackscited by 0
  17. 2026Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biasescited by 0
  18. 2026ToxScreen: Detecting Whether an LLM Has Been Poisonedcited by 0
  19. 2026Toward Secure AI-Powered Penetration Testing Agents: Security Threats, Guardrails, and Architectural Perspectivescited by 0