mlsanityData to 7 Oct 2026

Using Interleaved Ensemble Unlearning to Keep Backdoors at Bay for Finetuning Vision Transformers

arXiv 2410.01128 ↗First posted 1 Oct 2024cs.CV cs.LGcited by 1

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

Vision Transformers (ViTs) have become popular in computer vision tasks. Backdoor attacks, which trigger undesirable behaviours in models during inference, threaten ViTs' performance, particularly in security-sensitive tasks. Although backdoor defences have been developed for Convolutional Neural Networks (CNNs), they are less effective for ViTs, and defences tailored to ViTs are scarce. To address this, we present Interleaved Ensemble Unlearning (IEU), a method for finetuning clean ViTs on backdoored datasets. In stage 1, a shallow ViT is finetuned to have high confidence on backdoored data and low confidence on clean data. In stage 2, the shallow ViT acts as a ``gate'' to block potentially poisoned data from the defended ViT. This data is added to an unlearn set and asynchronously unlearned via gradient ascent. We demonstrate IEU's effectiveness on three datasets against 11 state-of-the-art backdoor attacks and show its versatility by applying it to different model architectures.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 48 of its 65 references that are in the map. First posted on arXiv 1 Oct 2024.

Builds on

48 of 65 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022TrojViT: Trojan Insertion in Vision Transformers27 Aug
  2. 2022Defending Backdoor Attacks on Vision Transformer via Patch Processing24 Jun
  3. 2021DBIA: Data-free Backdoor Injection Attack against Transformer Networks22 Nov
  4. 2021BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learning1 Aug
  5. 2021Rethinking the Backdoor Attacks' Triggers: A Frequency Perspective7 Apr

Other references · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2016Layer Normalizationcited by 5,579
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  5. 2016Wide Residual Networkscited by 3,057
  6. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  7. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  8. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  9. 2021Going deeper with Image Transformerscited by 759
  10. 2018Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networkscited by 550
  11. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  12. 2020Invisible Backdoor Attack with Sample-Specific Triggerscited by 338
31 more
  1. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  2. 2021Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networkscited by 313
  3. 2019Label-Consistent Backdoor Attackscited by 285
  4. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 276
  5. 2021Anti-Backdoor Learning: Training Clean Models on Poisoned Datacited by 258
  6. 2019A new Backdoor Attack in CNNs by training set corruption without label poisoningcited by 237
  7. 2021Adversarial Neuron Pruning Purifies Backdoored Deep Modelscited by 230
  8. 2020Blind Backdoors in Deep Learning Modelscited by 174
  9. 2021Adversarial Unlearning of Backdoors via Implicit Hypergradientcited by 160
  10. 2017Neural Trojanscited by 151
  11. 2020RAB: Provable Robustness Against Backdoor Attackscited by 115
  12. 2022Narcissus: A Practical Clean-Label Backdoor Attack with Limited Informationcited by 102
  13. 2021SiT: Self-supervised vIsion Transformercited by 99
  14. 2021Backdoor Attacks on Self-Supervised Learningcited by 89
  15. 2020Strong Data Augmentation Sanitizes Poisoning and Backdoor Attacks Without an Accuracy Tradeoffcited by 86
  16. 2023Semantic Segmentation using Vision Transformers: A surveycited by 73
  17. 2019Bypassing Backdoor Detection Algorithms in Deep Learningcited by 69
  18. 2023Reconstructive Neuron Pruning for Backdoor Defensecited by 56
  19. 2021Few-shot Backdoor Defense Using Shapley Estimationcited by 40
  20. 2022Training with More Confidence: Mitigating Injected and Natural Backdoors During Trainingcited by 38
  21. 2023From Shortcuts to Triggers: Backdoor Defense with Denoised PoEcited by 26
  22. 2022One-shot Neural Backdoor Erasing via Adversarial Weight Maskingcited by 25
  23. 2023BackdoorBox: A Python Toolbox for Backdoor Learningcited by 25
  24. 2023Object Detection with Transformers: A Reviewcited by 22
  25. 2023Defending Against Patch-based Backdoor Attacks on Self-Supervised Learningcited by 19
  26. 2024Backdoor Contrastive Learning via Bi-level Trigger Optimizationcited by 14
  27. 2022Trap and Replace: Defending Backdoor Attacks by Trapping Them into an Easy-to-Replace Subnetworkcited by 12
  28. 2022BATT: Backdoor Attack with Transformation-based Triggerscited by 9
  29. 2024Invisible Backdoor Attack against Self-supervised Learningcited by 5
  30. 2024A Survey of Vision Transformers in Autonomous Driving: Current Trends and Future Directionscited by 4
  31. 2024Clean-image Backdoor Attackscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Backdoor Directions in Vision Transformerscited by 1