mlsanityData to 7 Oct 2026

Solving Trojan Detection Competitions with Linear Weight Classification

arXiv 2411.03445 ↗First posted 5 Nov 2024cs.LG cs.AI cs.CLcited by 0

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

Neural networks can conceal malicious Trojan backdoors that allow a trigger to covertly change the model behavior. Detecting signs of these backdoors, particularly without access to any triggered data, is the subject of ongoing research and open challenges. In one common formulation of the problem, we are given a set of clean and poisoned models and need to predict whether a given test model is clean or poisoned. In this paper, we introduce a detector that works remarkably well across many of the existing datasets and domains. It is obtained by training a binary classifier on a large number of models' weights after performing a few different pre-processing steps including feature selection and standardization, reference model weights subtraction, and model alignment prior to detection. We evaluate this algorithm on a diverse set of Trojan detection benchmarks and domains and examine the cases where the approach is most and least effective.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 24 of its 35 references that are in the map. First posted on arXiv 5 Nov 2024.

Builds on

24 of 35 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Chain-of-Scrutiny: Detecting Backdoor Attacks for Large Language Models10 Jun
  2. 2021Trojan Signatures in DNN Weights7 Sep

Other references · 22

  1. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  2. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  3. 2018Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networkscited by 550
  4. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  5. 2019STRIP: A Defence Against Trojan Attacks on Deep Neural Networkscited by 437
  6. 2018Detecting Backdoor Attacks on Deep Neural Networks by Activation Clusteringcited by 429
  7. 2020Backdoor Learning: A Surveycited by 412
  8. 2019Hidden Trigger Backdoor Attackscited by 359
  9. 2019Detecting AI Trojans Using Meta Neural Analysiscited by 192
  10. 2018SentiNet: Detecting Localized Universal Attacks Against Deep Learning Systemscited by 148
  11. 2019Demon in the Variant: Statistical Analysis of DNNs for Robust Backdoor Contamination Detectioncited by 121
  12. 2019TABOR: A Highly Accurate Approach to Inspecting and Restoring Trojan Backdoors in AI Systemscited by 120
10 more
  1. 2021Backdoor Scanning for Deep Neural Networks through K-Arm Optimizationcited by 96
  2. 2020Practical Detection of Trojan Neural Networks: Data-Limited and Data-Free Casescited by 93
  3. 2020Can Adversarial Weight Perturbations Inject Neural Backdoors?cited by 51
  4. 2020The TrojAI Software Framework: An OpenSource tool for Embedding Trojans into Deep Learning Modelscited by 41
  5. 2022A Survey of Neural Trojan Attacks and Defenses in Deep Learningcited by 17
  6. 2020Poisoned classifiers are not only backdoored, they are fundamentally brokencited by 15
  7. 2021TOP: Backdoor Detection in Neural Networks via Transferability of Perturbationcited by 13
  8. 2020Odyssey: Creation, Analysis and Detection of Trojan Modelscited by 7
  9. 2020Noise-Response Analysis of Deep Neural Networks Quantifies Robustness and Fingerprints Structural Malwarecited by 7
  10. 2021MLDS: A Dataset for Weight-Space Analysis of Neural Networkscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Trojan Detection Through Pattern Recognition for Large Language Models
  2. 2025DISTIL: Data-Free Inversion of Suspicious Trojan Inputs via Latent Diffusion
  3. 2024DisDet: Exploring Detectability of Backdoor Attack on Diffusion Models
  4. 2025Lie Detector: Unified Backdoor Detection via Cross-Examination Framework
  5. 2026Trident: Improving Malware Detection with LLMs and Behavioral Features
  6. 2024Rethinking Backdoor Detection Evaluation for Language Models
  7. 2026Evaluating and Improving the Robustness of Large Language Models to Input Sequence Variations
  8. 2023Leveraging Diffusion-Based Image Variations for Robust Training on Poisoned Data