mlsanityData to 14 Oct 2026

Trojan Cleansing with Neural Collapse

arXiv 2411.12914 ↗First posted 19 Nov 2024cs.LG cs.CRcited by 1

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

Trojan attacks are sophisticated training-time attacks on neural networks that embed backdoor triggers which force the network to produce a specific output on any input which includes the trigger. With the increasing relevance of deep networks which are too large to train with personal resources and which are trained on data too large to thoroughly audit, these training-time attacks pose a significant risk. In this work, we connect trojan attacks to Neural Collapse, a phenomenon wherein the final feature representations of over-parameterized neural networks converge to a simple geometric structure. We provide experimental evidence that trojan attacks disrupt this convergence for a variety of datasets and architectures. We then use this disruption to design a lightweight, broadly generalizable mechanism for cleansing trojan attacks from a wide variety of different network architectures and experimentally demonstrate its efficacy.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 33 of its 44 references that are in the map. First posted on arXiv 19 Nov 2024.

Builds on

33 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Trojan Signatures in DNN Weights7 Sep

Other references · 32

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  3. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,055
  4. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  5. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  6. 2018Detecting Backdoor Attacks on Deep Neural Networks by Activation Clusteringcited by 456
  7. 2019Hidden Trigger Backdoor Attackscited by 371
  8. 2020Reflection Backdoor: A Natural Backdoor Attack on Deep Neural Networkscited by 349
  9. 2021Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networkscited by 324
  10. 2020Input-Aware Dynamic Backdoor Attackcited by 313
  11. 2019A new Backdoor Attack in CNNs by training set corruption without label poisoningcited by 242
  12. 2019A backdoor attack against LSTM-based text classification systemscited by 227
20 more
  1. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 227
  2. 2020BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvementscited by 211
  3. 2020Blind Backdoors in Deep Learning Modelscited by 182
  4. 2021Adversarial Unlearning of Backdoors via Implicit Hypergradientcited by 166
  5. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 158
  6. 2017Neural Trojanscited by 153
  7. 2022BackdoorBench: A Comprehensive Benchmark of Backdoor Learningcited by 128
  8. 2020An Embarrassingly Simple Approach for Trojan Attack in Deep Neural Networkscited by 97
  9. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  10. 2022BppAttack: Stealthy and Efficient Trojan Attacks against Deep Neural Networks via Image Quantization and Contrastive Adversarial Learningcited by 86
  11. 2023Enhancing Fine-Tuning Based Backdoor Defense with Sharpness-Aware Minimizationcited by 64
  12. 2022The "Beatrix'' Resurrections: Robust Backdoor Detection via Gram Matricescited by 63
  13. 2023Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Datacited by 38
  14. 2023Neural Polarizer: A Lightweight and Effective Backdoor Defense via Purifying Poisoned Featurescited by 37
  15. 2021Topological Detection of Trojaned Neural Networkscited by 34
  16. 2022An Embarrassingly Simple Backdoor Attack on Self-supervised Learningcited by 31
  17. 2020CLEANN: Accelerated Trojan Shield for Embedded Neural Networkscited by 22
  18. 2022Eliminating Backdoor Triggers for Deep Neural Networks Using Attention Relation Graph Distillationcited by 18
  19. 2024Cross Entropy versus Label Smoothing: A Neural Collapse Perspectivecited by 11
  20. 2021Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equationscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Thinking Wrong in Silence: Backdoor Attacks on Continuous Latent Reasoningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2022Dormant Neural Trojans
  2. 2022Quarantine: Sparsity Can Uncover the Trojan Attack Trigger for Free
  3. 2022An Adaptive Black-box Backdoor Detection Method for Deep Neural Networks
  4. 2021CatchBackdoor: Backdoor Detection via Critical Trojan Neural Path Fuzzing
  5. 2022An Adaptive Black-box Defense against Trojan Attacks (TrojDef)
  6. 2022Rethinking the Reverse-engineering of Trojan Triggers
  7. 2024Solving Trojan Detection Competitions with Linear Weight Classification
  8. 2023Backdoor Mitigation by Correcting the Distribution of Neural Activations