# Concealed Data Poisoning Attacks on NLP Models

arXiv: https://arxiv.org/abs/2010.12563 · HTML: https://mlsanity.com/paper/2010.12563/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Oct 2020.

## Built on by (most cited first, 14 of 14)

- [Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections](https://mlsanity.com/paper/2104.04670/) (2021)
- [A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment](https://mlsanity.com/paper/2504.15585/) (2025)
- [TrojDiff: Trojan Attacks on Diffusion Models with Diverse Targets](https://mlsanity.com/paper/2303.05762/) (2023)
- [Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasures](https://mlsanity.com/paper/2112.05224/) (2021)
- [Putting words into the system's mouth: A targeted attack on neural machine translation using monolingual data poisoning](https://mlsanity.com/paper/2107.05243/) (2021)
- [A Survey on Responsible Generative AI: What to Generate and What Not](https://mlsanity.com/paper/2404.05783/) (2024)
- [A Systematic Review of Poisoning Attacks Against Large Language Models](https://mlsanity.com/paper/2506.06518/) (2025)
- [Spinning Sequence-to-Sequence Models with Meta-Backdoors](https://mlsanity.com/paper/2107.10443/) (2021)
- [Learned-Database Systems Security](https://mlsanity.com/paper/2212.10318/) (2022)
- [LoRA Users Beware: A Few Spurious Tokens Can Manipulate Your Finetuned Model](https://mlsanity.com/paper/2506.11402/) (2025)
- [Mithridates: Auditing and Boosting Backdoor Resistance of Machine Learning Pipelines](https://mlsanity.com/paper/2302.04977/) (2023)
- [Task-Agnostic Language Model Watermarking via High Entropy Passthrough Layers](https://mlsanity.com/paper/2412.12563/) (2024)
- [The Persistent Vulnerability of Aligned AI Systems](https://mlsanity.com/paper/2604.00324/) (2026)
- [PermaFrost-Attack: Stealth Pretraining Seeding(SPS) for planting Logic Landmines During LLM Training](https://mlsanity.com/paper/2604.22117/) (2026)

## Abstract (arXiv)

Adversarial attacks alter NLP model predictions by perturbing test-time inputs. However, it is much less understood whether, and how, predictions can be manipulated with small, concealed changes to the training data. In this work, we develop a new data poisoning attack that allows an adversary to control model predictions whenever a desired trigger phrase is present in the input. For instance, we insert 50 poison examples into a sentiment model's training set that causes the model to frequently predict Positive whenever the input contains "James Bond". Crucially, we craft these poison examples using a gradient-based procedure so that they do not mention the trigger phrase. We also apply our poison attack to language modeling ("Apple iPhone" triggers negative generations) and machine translation ("iced coffee" mistranslated as "hot coffee"). We conclude by proposing three defenses that can mitigate our attack at some cost in prediction accuracy or extra human annotation.