# ParaFuzz: An Interpretability-Driven Technique for Detecting Poisoned Samples in NLP

arXiv: https://arxiv.org/abs/2308.02122 · HTML: https://mlsanity.com/paper/2308.02122/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security.
- Builds on 29 of its 43 references that are in the map.
- First posted on arXiv 4 Aug 2023.

## Placed in

- LLM safety and security / [Backdoor and data poisoning attacks](https://mlsanity.com/safety/backdoor-and-data-poisoning-attacks/)

## Builds on (most cited first, 29 of 29)

- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense](https://mlsanity.com/paper/2303.13408/) (2023)
- [Can AI-Generated Text be Reliably Detected?](https://mlsanity.com/paper/2303.11156/) (2023)
- [Weight Poisoning Attacks on Pre-trained Models](https://mlsanity.com/paper/2004.06660/) (2020)
- [ONION: A Simple and Effective Defense Against Textual Backdoor Attacks](https://mlsanity.com/paper/2011.10369/) (2020)
- [A backdoor attack against LSTM-based text classification systems](https://mlsanity.com/paper/1905.12457/) (2019)
- [Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger](https://mlsanity.com/paper/2105.12400/) (2021)
- [BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvements](https://mlsanity.com/paper/2006.01043/) (2020)
- [Detecting AI Trojans Using Meta Neural Analysis](https://mlsanity.com/paper/1910.03137/) (2019)
- [Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer](https://mlsanity.com/paper/2110.07139/) (2021)
- [The Science of Detecting LLM-Generated Texts](https://mlsanity.com/paper/2303.07205/) (2023)
- [Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoning](https://mlsanity.com/paper/2108.13888/) (2021)
- [Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Models](https://mlsanity.com/paper/2103.15543/) (2021)
- [RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Models](https://mlsanity.com/paper/2110.07831/) (2021)
- [Trojaning Language Models for Fun and Profit](https://mlsanity.com/paper/2008.00312/) (2020)
- [Hidden Backdoors in Human-Centric Language Models](https://mlsanity.com/paper/2105.00164/) (2021)
- [Turn the Combination Lock: Learnable Textual Backdoor Attacks via Word Substitution](https://mlsanity.com/paper/2106.06361/) (2021)
- [Mitigating backdoor attacks in LSTM-based Text Classification Systems by Backdoor Keyword Identification](https://mlsanity.com/paper/2007.12070/) (2020)
- [Backdoor Pre-trained Models Can Transfer to All](https://mlsanity.com/paper/2111.00197/) (2021)
- [A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks](https://mlsanity.com/paper/2206.08514/) (2022)
- [Design and Evaluation of a Multi-Domain Trojan Detection Method on Deep Neural Networks](https://mlsanity.com/paper/1911.10312/) (2019)
- [T-Miner: A Generative Approach to Defend Against Trojan Attacks on DNN-based Text Classification](https://mlsanity.com/paper/2103.04264/) (2021)
- [Exploring the Universal Vulnerability of Prompt-based Learning Paradigm](https://mlsanity.com/paper/2204.05239/) (2022)
- [Can Adversarial Weight Perturbations Inject Neural Backdoors?](https://mlsanity.com/paper/2008.01761/) (2020)
- [Red Alarm for Pre-trained Models: Universal Vulnerability to Neuron-Level Backdoor Attacks](https://mlsanity.com/paper/2101.06969/) (2021)
- [Constrained Optimization with Dynamic Bound-scaling for Effective NLPBackdoor Defense](https://mlsanity.com/paper/2202.05749/) (2022)
- [Limits of Detecting Text Generated by Large-Scale Language Models](https://mlsanity.com/paper/2002.03438/) (2020)
- [A Survey on Backdoor Attack and Defense in Natural Language Processing](https://mlsanity.com/paper/2211.11958/) (2022)

## Built on by (most cited first, 19 of 19)

- [Security and Privacy Challenges of Large Language Models: A Survey](https://mlsanity.com/paper/2402.00888/) (2024)
- [The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies](https://mlsanity.com/paper/2407.19354/) (2024)
- [When LLM Meets DRL: Advancing Jailbreaking Efficiency via DRL-guided Search](https://mlsanity.com/paper/2406.08705/) (2024)
- [Backdoor Attacks and Countermeasures in Natural Language Processing Models: A Comprehensive Security Review](https://mlsanity.com/paper/2309.06055/) (2023)
- [Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Models](https://mlsanity.com/paper/2506.16447/) (2025)
- [CLIBE: Detecting Dynamic Backdoors in Transformer-based NLP Models](https://mlsanity.com/paper/2409.01193/) (2024)
- [SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attacks](https://mlsanity.com/paper/2506.10424/) (2025)
- [PromptPex: Automatic Test Generation for Language Model Prompts](https://mlsanity.com/paper/2503.05070/) (2025)
- [Concept-Level Explainability for Auditing & Steering LLM Responses](https://mlsanity.com/paper/2505.07610/) (2025)
- [TrojFM: Resource-efficient Backdoor Attacks against Very Large Foundation Models](https://mlsanity.com/paper/2405.16783/) (2024)
- [On the Challenges of Fuzzing Techniques via Large Language Models](https://mlsanity.com/paper/2402.00350/) (2024)
- [Temporal Logic-Based Multi-Vehicle Backdoor Attacks against Offline RL Agents in End-to-end Autonomous Driving](https://mlsanity.com/paper/2509.16950/) (2025)
- [Neuroplasticity and Corruption in Model Mechanisms: A Case Study Of Indirect Object Identification](https://mlsanity.com/paper/2503.01896/) (2025)
- [Lethe: Purifying Backdoored Large Language Models with Knowledge Dilution](https://mlsanity.com/paper/2508.21004/) (2025)
- [SPARTA: Evaluating Reasoning Segmentation Robustness through Black-Box Adversarial Paraphrasing in Text Autoencoder Latent Space](https://mlsanity.com/paper/2510.24446/) (2025)
- [MemoPhishAgent: Memory-Augmented Multi-Modal LLM Agent for Phishing URL Detection](https://mlsanity.com/paper/2602.21394/) (2026)
- [Large Language Models in Cybersecurity: Applications, Vulnerabilities, and Defense Techniques](https://mlsanity.com/paper/2507.13629/) (2025)
- [LLMs are All You Need? Improving Fuzz Testing for MOJO with Large Language Models](https://mlsanity.com/paper/2510.10179/) (2025)
- [Turn-Based Structural Triggers: Structure-Conditioned Backdoors in Multi-Turn LLMs](https://mlsanity.com/paper/2601.14340/) (2026)

## Abstract (arXiv)

Backdoor attacks have emerged as a prominent threat to natural language processing (NLP) models, where the presence of specific triggers in the input can lead poisoned models to misclassify these inputs to predetermined target classes. Current detection mechanisms are limited by their inability to address more covert backdoor strategies, such as style-based attacks. In this work, we propose an innovative test-time poisoned sample detection framework that hinges on the interpretability of model predictions, grounded in the semantic meaning of inputs. We contend that triggers (e.g., infrequent words) are not supposed to fundamentally alter the underlying semantic meanings of poisoned samples as they want to stay stealthy. Based on this observation, we hypothesize that while the model's predictions for paraphrased clean samples should remain stable, predictions for poisoned samples should revert to their true labels upon the mutations applied to triggers during the paraphrasing process. We employ ChatGPT, a state-of-the-art large language model, as our paraphraser and formulate the trigger-removal task as a prompt engineering problem. We adopt fuzzing, a technique commonly used for unearthing software vulnerabilities, to discover optimal paraphrase prompts that can effectively eliminate triggers while concurrently maintaining input semantics. Experiments on 4 types of backdoor attacks, including the subtle style backdoors, and 4 distinct datasets demonstrate that our approach surpasses baseline methods, including STRIP, RAP, and ONION, in precision and recall.