# Humpty Dumpty: Controlling Word Meanings via Corpus Poisoning

arXiv: https://arxiv.org/abs/2001.04935 · HTML: https://mlsanity.com/paper/2001.04935/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jan 2020.

## Built on by (most cited first, 18 of 18)

- [Poisoning Retrieval Corpora by Injecting Adversarial Passages](https://mlsanity.com/paper/2310.19156/) (2023)
- [Universal Jailbreak Backdoors from Poisoned Human Feedback](https://mlsanity.com/paper/2311.14455/) (2023)
- [Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Models](https://mlsanity.com/paper/2103.15543/) (2021)
- [Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasures](https://mlsanity.com/paper/2112.05224/) (2021)
- ["Real Attackers Don't Compute Gradients": Bridging the Gap Between Adversarial ML Research and Practice](https://mlsanity.com/paper/2212.14315/) (2022)
- [Red Alarm for Pre-trained Models: Universal Vulnerability to Neuron-Level Backdoor Attacks](https://mlsanity.com/paper/2101.06969/) (2021)
- [Persistent Pre-Training Poisoning of LLMs](https://mlsanity.com/paper/2410.13722/) (2024)
- [PoisonedEncoder: Poisoning the Unlabeled Pre-training Data in Contrastive Learning](https://mlsanity.com/paper/2205.06401/) (2022)
- [Competition Report: Finding Universal Jailbreak Backdoors in Aligned LLMs](https://mlsanity.com/paper/2404.14461/) (2024)
- [Data Poisoning for In-context Learning](https://mlsanity.com/paper/2402.02160/) (2024)
- [Corpus Poisoning via Approximate Greedy Gradient Descent](https://mlsanity.com/paper/2406.05087/) (2024)
- [Backdoor Attacks on Dense Retrieval via Public and Unintentional Triggers](https://mlsanity.com/paper/2402.13532/) (2024)
- [Manipulating Transfer Learning for Property Inference](https://mlsanity.com/paper/2303.11643/) (2023)
- [Rethinking Membership Inference Attacks Against Transfer Learning](https://mlsanity.com/paper/2501.11577/) (2025)
- [Learned-Database Systems Security](https://mlsanity.com/paper/2212.10318/) (2022)
- [Dropout Attacks](https://mlsanity.com/paper/2309.01614/) (2023)
- [Attacks against Ranking Algorithms with Text Embeddings: a Case Study on Recruitment Algorithms](https://mlsanity.com/paper/2108.05490/) (2021)
- [Manipulating Predictions over Discrete Inputs in Machine Teaching](https://mlsanity.com/paper/2401.17865/) (2024)

## Abstract (arXiv)

Word embeddings, i.e., low-dimensional vector representations such as GloVe and SGNS, encode word "meaning" in the sense that distances between words' vectors correspond to their semantic proximity. This enables transfer learning of semantics for a variety of natural language processing tasks. Word embeddings are typically trained on large public corpora such as Wikipedia or Twitter. We demonstrate that an attacker who can modify the corpus on which the embedding is trained can control the "meaning" of new and existing words by changing their locations in the embedding space. We develop an explicit expression over corpus features that serves as a proxy for distance between words and establish a causative relationship between its values and embedding distances. We then show how to use this relationship for two adversarial objectives: (1) make a word a top-ranked neighbor of another word, and (2) move a word from one semantic cluster to another. An attack on the embedding can affect diverse downstream tasks, demonstrating for the first time the power of data poisoning in transfer learning scenarios. We use this attack to manipulate query expansion in information retrieval systems such as resume search, make certain names more or less visible to named entity recognition models, and cause new words to be translated to a particular target word regardless of the language. Finally, we show how the attacker can generate linguistically likely corpus modifications, thus fooling defenses that attempt to filter implausible sentences from the corpus using a language model.