mlsanityData to 7 Oct 2026

Analyzing And Editing Inner Mechanisms Of Backdoored Language Models

arXiv 2302.12461 ↗First posted 24 Feb 2023cs.LG cs.AI cs.CLcited by 22

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

Poisoning of data sets is a potential security threat to large language models that can lead to backdoored models. A description of the internal mechanisms of backdoored language models and how they process trigger inputs, e.g., when switching to toxic language, has yet to be found. In this work, we study the internal representations of transformer-based backdoored language models and determine early-layer MLP modules as most important for the backdoor mechanism in combination with the initial embedding projection. We use this knowledge to remove, insert, and modify backdoor mechanisms with engineered replacements that reduce the MLP module outputs to essentials for the backdoor mechanism. To this end, we introduce PCP ablation, where we replace transformer modules with low-rank matrices based on the principal components of their activations. We demonstrate our results on backdoored toy, backdoored large, and non-backdoored open-source models. We show that we can improve the backdoor robustness of large language models by locally constraining individual modules during fine-tuning on potentially poisonous data sets. Trigger warning: Offensive language.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 48 of its 82 references that are in the map. First posted on arXiv 24 Feb 2023.

Builds on

48 of 82 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Backdoor Attacks on Time Series: A Generative Approach15 Nov
  2. 2021Triggerless Backdoor Attack for NLP Tasks with Clean Labels15 Nov

Other references · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  5. 2022Emergent Abilities of Large Language Modelscited by 2,483
  6. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  7. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  8. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  9. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  10. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  11. 2022Red Teaming Language Models with Language Modelscited by 983
  12. 2022In-context Learning and Induction Headscited by 930
34 more
  1. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  2. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  3. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  4. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  5. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  6. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  7. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  8. 2020Backdoor Learning: A Surveycited by 412
  9. 2020Weight Poisoning Attacks on Pre-trained Modelscited by 318
  10. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 294
  11. 2021Unsolved Problems in ML Safetycited by 291
  12. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 276
  13. 2017Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypescited by 251
  14. 2021Adversarial Neuron Pruning Purifies Backdoored Deep Modelscited by 230
  15. 2020Towards Debiasing Sentence Representationscited by 170
  16. 2018Learning Gender-Neutral Word Embeddingscited by 169
  17. 2017Learning to Generate Reviews and Discovering Sentimentcited by 147
  18. 2019Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddingscited by 146
  19. 2022Analyzing Transformers in Embedding Spacecited by 131
  20. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  21. 2020Adversarial Machine Learning -- Industry Perspectivescited by 121
  22. 2021Inducing Causal Structure for Interpretable Neural Networkscited by 107
  23. 2021Backdoor Attacks on Self-Supervised Learningcited by 89
  24. 2020Trojaning Language Models for Fun and Profitcited by 80
  25. 2022Understanding Transformer Memorization Recall Through Idiomscited by 62
  26. 2022Planting Undetectable Backdoors in Machine Learning Modelscited by 55
  27. 2017Simple and Effective Dimensionality Reduction for Word Embeddingscited by 38
  28. 2019Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamicscited by 38
  29. 2019Are We Consistently Biased? Multidimensional Analysis of Biases in Distributional Word Vectorscited by 31
  30. 2020Deep Learning Backdoorscited by 19
  31. 2020VenoMave: Targeted Poisoning Against Speech Recognitioncited by 7
  32. 2022Reducing Certified Regression to Certified Classification for General Poisoning Attackscited by 7
  33. 2021Spinning Sequence-to-Sequence Models with Meta-Backdoorscited by 5
  34. 2022Sniper Backdoor: Single Client Targeted Backdoor Attack in Federated Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 5; 2025: 4; 2026: 9.

22 papers.

Sort:

By area:

  1. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  2. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 367
  3. 2023Jump to Conclusions: Short-Cutting Transformers With Linear Transformationscited by 89
  4. 2023VISIT: Visualizing and Interpreting the Semantic Information Flow of Transformerscited by 32
  5. 2024Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementationcited by 27
  6. 2024Here's a Free Lunch: Sanitizing Backdoored Models with Model Mergecited by 21
  7. 2025Depth Gives a False Sense of Privacy: LLM Internal States Inversioncited by 14
  8. 2025Unlearning Isn't Invisible: Detecting Unlearning Traces in LLMs from Model Outputscited by 10
  9. 2025Cross-Modal Unlearning via Influential Neuron Path Editing in Multimodal Large Language Modelscited by 7
  10. 2024Acquiring Clean Language Models from Backdoor Poisoned Datasets by Downscaling Frequency Spacecited by 6
  11. 2026Purifying Generative LLMs from Backdoors without Prior Knowledge or Clean Referencecited by 6
  12. 2024Markovian Transformers for Informative Language Modelingcited by 4
10 more
  1. 2026One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Modelscited by 3
  2. 2023HANS, are you clever? Clever Hans Effect Analysis of Neural Systemscited by 2
  3. 2025Password-Activated Shutdown Protocols for Misaligned Frontier Agentscited by 0
  4. 2026ProjLens: Unveiling the Role of Projectors in Multimodal Model Safetycited by 0
  5. 2026Backdoor Channels Hidden in Latent Space: Extending Cryptographic Undetectability to Modern Neural Networkscited by 0
  6. 2026Backdoor Unlearning Generalization: A Path Toward the Removal of Unknown Triggers in LLMscited by 0
  7. 2026Curvature-Guided Module Localization for Low-Rank Detoxification of Backdoored Large Language Modelscited by 0
  8. 2026Removing the NEEDLE in the Haystack: Backdoor Removal in LLMs via Weight Orthogonalisationcited by 0
  9. 2026Backdoor Containment via Expert Quarantine and Shutdown in LLMscited by 0
  10. 2026Backdoor Purification for LoRA-Tuned LLMs via Null-Space Projectioncited by 0