mlsanityData to 7 Oct 2026

Diffusion Theory as a Scalpel: Detecting and Purifying Poisonous Dimensions in Pre-trained Language Models Caused by Backdoor or Bias

arXiv 2305.04547 ↗First posted 8 May 2023cs.CLcited by 7

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

Pre-trained Language Models (PLMs) may be poisonous with backdoors or bias injected by the suspicious attacker during the fine-tuning process. A core challenge of purifying potentially poisonous PLMs is precisely finding poisonous dimensions. To settle this issue, we propose the Fine-purifying approach, which utilizes the diffusion theory to study the dynamic process of fine-tuning for finding potentially poisonous dimensions. According to the relationship between parameter drifts and Hessians of different dimensions, we can detect poisonous dimensions with abnormal dynamics, purify them by resetting them to clean pre-trained weights, and then fine-tune the purified weights on a small clean dataset. To the best of our knowledge, we are the first to study the dynamics guided by the diffusion theory for safety or defense purposes. Experimental results validate the effectiveness of Fine-purifying even with a small clean dataset.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 39 of its 58 references that are in the map. First posted on arXiv 8 May 2023.

Builds on

39 of 58 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Fine-mixing: Mitigating Backdoors in Fine-tuned Language Models18 Oct
  2. 2021RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Models15 Oct
  3. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger26 May
  4. 2021Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Models29 Mar

Other references · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  12. 2018Deep contextualized word representationscited by 2,285
23 more
  1. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  2. 2018Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networkscited by 550
  3. 2019STRIP: A Defence Against Trojan Attacks on Deep Neural Networkscited by 437
  4. 2020Weight Poisoning Attacks on Pre-trained Modelscited by 318
  5. 2021Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networkscited by 313
  6. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
  7. 2017Overcoming Catastrophic Forgetting by Incremental Moment Matchingcited by 243
  8. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 231
  9. 2019A backdoor attack against LSTM-based text classification systemscited by 214
  10. 2017Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimizationcited by 188
  11. 2020Clean-Label Backdoor Attacks on Video Recognition Modelscited by 162
  12. 2013Revisiting Natural Gradient for Deep Networkscited by 157
  13. 2019Distributionally Robust Language Modelingcited by 142
  14. 2020Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustnesscited by 130
  15. 2020Towards Debiasing NLU Models from Unknown Biasescited by 116
  16. 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 109
  17. 2021Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoningcited by 103
  18. 2020Mitigating backdoor attacks in LSTM-based Text Classification Systems by Backdoor Keyword Identificationcited by 80
  19. 2018FNNC: Achieving Fairness through Neural Networkscited by 43
  20. 2022Few-Shot Backdoor Attacks on Visual Object Trackingcited by 35
  21. 2021Modeling the Second Player in Distributionally Robust Optimizationcited by 29
  22. 2021How to Inject Backdoors with Better Consistency: Logit Anchoring on Clean Datacited by 22
  23. 2020Machine Learning for Offensive Security: Sandbox Classification Using Decision Trees and Artificial Neural Networkscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 1; 2025: 3; 2026: 1.

7 papers.

  1. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 59
  2. 2024Fairness in Large Language Models: A Taxonomic Surveycited by 57
  3. 2025Pruning Strategies for Backdoor Defense in LLMscited by 4
  4. 2025Unmasking Backdoors: An Explainable Defense via Gradient-Attention Anomaly Scoring for Pre-trained Language Modelscited by 3
  5. 2023Trustworthy Large Models in Vision: A Surveycited by 0
  6. 2025Backdoor Samples Detection Based on Perturbation Discrepancy Consistency in Pre-trained Language Modelscited by 0
  7. 2026Trojan horse hunt in deep forecasting models: Insights from the European Space Agency competitioncited by 0