mlsanityData to 7 Oct 2026

Towards Understanding Dual BN In Hybrid Adversarial Training

arXiv 2403.19150 ↗First posted 28 Mar 2024cs.LG cs.AI cs.CRcited by 0

Not on the map. Why?

Abstract (arXiv)

There is a growing concern about applying batch normalization (BN) in adversarial training (AT), especially when the model is trained on both adversarial samples and clean samples (termed Hybrid-AT). With the assumption that adversarial and clean samples are from two different domains, a common practice in prior works is to adopt Dual BN, where BN and BN are used for adversarial and clean branches, respectively. A popular belief for motivating Dual BN is that estimating normalization statistics of this mixture distribution is challenging and thus disentangling it for normalization achieves stronger robustness. In contrast to this belief, we reveal that disentangling statistics plays a less role than disentangling affine parameters in model training. This finding aligns with prior work (Rebuffi et al., 2023), and we build upon their research for further investigations. We demonstrate that the domain gap between adversarial and clean samples is not very large, which is counter-intuitive considering the significant influence of adversarial perturbation on the model accuracy. We further propose a two-task hypothesis which serves as the empirical foundation and a unified framework for Hybrid-AT improvement. We also investigate Dual BN in test-time and reveal that affine parameters characterize the robustness during inference. Overall, our work sheds new light on understanding the mechanism of Dual BN in Hybrid-AT and its underlying justification.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 46 of its 55 references that are in the map. First posted on arXiv 28 Mar 2024.

Builds on

46 of 55 references.

References · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2016Densely Connected Convolutional Networkscited by 6,691
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  7. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  8. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  9. 2020Fast is better than free: Revisiting adversarial trainingcited by 723
  10. 2020Improving robustness against common corruptions by covariate shift adaptationcited by 427
  11. 2019Unlabeled Data Improves Adversarial Robustnesscited by 415
  12. 2018How Does Batch Normalization Help Optimization?cited by 355
34 more
  1. 2016Revisiting Batch Normalization For Practical Domain Adaptationcited by 285
  2. 2019Adversarial Examples Improve Image Recognitioncited by 283
  3. 2021Recent Advances in Adversarial Training for Adversarial Robustnesscited by 276
  4. 2020Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examplescited by 275
  5. 2018Adversarial Logit Pairingcited by 224
  6. 2020Understanding and Improving Fast Adversarial Trainingcited by 213
  7. 2020Bag of Tricks for Adversarial Trainingcited by 199
  8. 2019Domain-Specific Batch Normalization for Unsupervised Domain Adaptationcited by 197
  9. 2016Wider or Deeper: Revisiting the ResNet Model for Visual Recognitioncited by 184
  10. 2019Are Labels Required for Improving Adversarial Robustness?cited by 162
  11. 2018Understanding Batch Normalizationcited by 157
  12. 2016Correlation Alignment for Unsupervised Domain Adaptationcited by 146
  13. 2020Robust Pre-Training by Adversarial Contrastive Learningcited by 144
  14. 2016Resnet in Resnet: Generalizing Residual Architecturescited by 110
  15. 2020Smooth Adversarial Trainingcited by 89
  16. 2021AugMax: Adversarial Composition of Random Augmentations for Robust Trainingcited by 89
  17. 2021When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?cited by 76
  18. 2021A Survey On Universal Adversarial Attackcited by 59
  19. 2020Revisiting Batch Normalization for Improving Corruption Robustnesscited by 55
  20. 2020Once-for-All Adversarial Training: In-Situ Tradeoff between Robustness and Accuracy for Freecited by 48
  21. 2022Make Some Noise: Reliable and Efficient Single-Step Adversarial Trainingcited by 42
  22. 2021Boosting Fast Adversarial Training with Learnable Adversarial Initializationcited by 40
  23. 2020Batch Normalization Increases Adversarial Vulnerability and Decreases Adversarial Transferability: A Non-Robust Feature Perspectivecited by 35
  24. 2022Prior-Guided Adversarial Initialization for Fast Adversarial Trainingcited by 34
  25. 2021Reliably fast adversarial training via latent adversarial perturbationcited by 29
  26. 2022Decoupled Adversarial Contrastive Learning for Self-supervised Adversarial Robustnesscited by 20
  27. 2022Revisiting adapters with adversarial trainingcited by 18
  28. 2020ResNet or DenseNet? Introducing Dense Shortcuts to ResNetcited by 12
  29. 2020Adversarial Momentum-Contrastive Pre-Trainingcited by 10
  30. 2020Towards an Adversarially Robust Normalization Approachcited by 9
  31. 2021Sandwich Batch Normalization: A Drop-In Replacement for Feature Distribution Heterogeneitycited by 7
  32. 2016Demystifying ResNetcited by 4
  33. 2019Where is the Bottleneck of Adversarial Learning with Unlabeled Data?cited by 3
  34. 2021Reciprocal Normalization for Domain Adaptationcited by 3

Built on by

Not cited yet.